Remaining Useful Life Prediction of Hybrid Ceramic Bearings Using an Integrated Deep Learning and Particle Filter Approach

Abstract

Bearings are one of the most critical components in many industrial machines. Predicting remaining useful life (RUL) of bearings has been an important task for condition-based maintenance of industrial machines. One critical challenge for performing such tasks in the era of the Internet of Things and Industrial 4.0, is to automatically process massive amounts of data and accurately predict the RUL of bearings. This paper addresses the limitations of traditional data-driven prognostics, and presents a new method that integrates a deep belief network and a particle filter for RUL prediction of hybrid ceramic bearings. Real data collected from hybrid ceramic bearing run-to-failure tests were used to test and validate the integrated method. The performance of the integrated method was also compared with deep belief network and particle filter-based approaches. The validation and comparison results showed that RUL prediction performance using the integrated method was promising.

1. Introduction

Mechanical big data has the characteristics of large volume, diversity, and high velocity. The prediction of remaining useful life (RUL) has been used as an important parameter for condition-based maintenance decision making [1]. Traditional data-driven prognostic methods have their limitations for processing mechanical big data. Many prognostic methods are developed based on explicit model equations [2]. Therefore, they are largely dependent on human expertise and knowledge in model building and signal processing, and hence are difficult to implement for automatic processing of massive data. These prognostic methods include recurrent neural networks [3,4], Kalman filters [5,6,7], dynamic Bayesian networks [8], k-reliable decentralized prognosis [9], particle filter-based [10,11,12,13], and combined particle filter and neural networks [14]. In addition, some fuzzy systems-based approaches for prediction have been developed [15]. However, in comparison with fuzzy systems, particle filters take a probabilistic approach, in that the posterior distribution is modeled by sampling from a set of distributions, whereas in fuzzy systems the model output is based on the input variables of fuzzy set membership and an implication of rules. Among all the approaches, particle filters [16] have emerged in recent years as a comparatively good RUL prediction method and are becoming more and more widespread, mainly due to their capability of dealing with dynamic systems characterized with nonlinear and non-Gaussian natures. For example, Yoon and He [17] showed the superior RUL prediction performance of a particle filter-based approach using the gear data provided by the NASA Glenn Spiral Bevel Gear Test Facility.

However, all of the above mentioned methods require either complicated signal processing techniques to extract features from the sensor data, or knowledge of system dynamics to build the explicit model equations. This type of requirement involves manual processing and analysis of data by human experts, and therefore makes these methods unsuitable for automatic data processing and feature extraction for big data. There also may be situations in which these models are unavailable, such as in offshore well drilling and wind turbines, or for some bearings in which online measurements of the damage may not be available, in which the traditional particle filter approach cannot be used [11]. To effectively extract features from massive bearing condition monitoring data and accurately predict bearing RUL, new effective methods are needed. The recent developments in deep learning have provided an attractive opportunity to build advanced RUL prediction methods for big data.

Since the introduction of a deep belief network [18], deep belief networks and other deep learning methods have become widely used for big data processing and analysis. Deep learning is capable of extracting useful and important features from data to improve the power of prediction [19]. Equipped with multiple layers of structure, deep learning is also capable of processing massive data and extracting hidden information. One recent successful story of deep learning is AlphaGo by Google Deepmind [20]. AlphaGo has demonstrated the power of deep learning for massive data processing and feature learning by defeating the best human Go player in the world. Deep neural network architectures have been successfully built in various domains such as image recognition, automatic speech recognition, and natural language processing [21]. Recently [22,23] applied deep learning on raw vibration signals and time-domain features for machine fault diagnostics. In addition to classification problems, deep learning also has promising capability to solve prediction problems. These prediction problems include predicting car traffic [24], weather [25], wind speed [26], and internet traffic [27]. A number of deep learning algorithms have been used for solving prediction problems. These include auto-encoders, restricted Boltzmann machines, deep belief networks, convolutional neural networks, and more. Deep learning represents an attractive option to process massive data for RUL prediction. Deutsch and He [28] presented a deep learning-based bearing RUL prediction method using a deep brief network, and compared RUL prediction performance with a particle filter-based approach. Their comparison results showed that even though the RUL prediction performance of the deep learning based approach was comparable with particle filter-based approach, the RUL prediction accuracy of the deep learning-based approach was slightly lower than that of the particle filter. Therefore, it was of interest to investigate an integrated method that can use the strength of deep learning to overcome the limitations of the particle filter. To date, no research on combining deep learning with particle filter for RUL prediction of hybrid ceramic bearings has been reported in the literature.

In this paper, a new integrated method that combines a deep belief network with a particle filter for remaining useful life prediction of hybrid ceramic bearings using vibration signals was presented. Real vibration data collected from hybrid ceramic bearing run-to-failure tests were used to test and validate the integrated method. The performance of the integrated method was also compared with a deep belief network and particle filter-based approaches.

2. Methodology

A deep belief network (DBN), which is a stacked version of a restricted Boltzmann machine (RBM), and a purely data-driven particle filter were combined in this paper in order to predict the RUL at L steps ahead into the future. Next, the relevant background of RBM, DBN and the particle filter was provided. The section concludes with the combined DBN and particle filter-based approach for RUL prediction.

2.1. The Restricted Boltzmann Machine

An RBM [29] is considered as a type of unsupervised machine learning method. It is a stochastic artificial neural network that learns a probability distribution over the set of its inputs. A RBM normally has two layers: a visible layer and a hidden layer. It can be represented by a bipartite graph that contains undirected edges from its two layers. Each layer contains a collection of neurons/nodes. Each neuron/node of the visible layer represents a feature of the input data, while neurons/nodes of the hidden layer represent the latent variables. A typical RBM structure is shown in Figure 1. The reason that an RBM is “Restricted” is because there are no connections between each neuron/node within either the visible or hidden layers. An RBM contains a matrix of weights $W_{ij}$ representing the connection to visible node $v_i$ and hidden node $h_j$. In Figure 1, $a_i$ represents the bias term in the visible layer, and $b_j$ in the hidden layer.

The weights and biases are computed by maximizing $P\left(\mathit{v}\right)$, the probability that the network assigns to a visible vector $\mathit{v}$:

$$P\left(\mathit{v}\right)=\frac{1}{Z}{\displaystyle \sum _{\mathit{h}}}{e}^{-E\left(\mathit{v},\mathit{h}\right)}$$

(1)

where $Z$ is the normalization constant that can be obtained by summing over all the possible pairs of visible and hidden vectors:

$$Z={\displaystyle \sum _{\mathit{v}}\sum _{\mathit{h}}}{e}^{-E\left(\mathit{v},\mathit{h}\right)}$$

(2)

and the energy function of the joint configuration $\left(\mathit{v},\mathit{h}\right)$ is given by:

$$E\left(\mathit{v},\mathit{h}\right)=-{\mathit{a}}^T\mathit{v}-{\mathit{b}}^T\mathit{h}-{\mathit{v}}^T\mathit{W}\mathit{h}$$

(3)

Theoretically, the problem of maximizing Equation (1) can be solved by taking its partial log derivative with respect to its parameters $\mathit{W}$, $\mathit{a},$ and $\mathit{b}$:

$$\frac{\partial \log \left(P\left(\mathit{v}\right)\right)}{\partial \mathit{W},\mathit{a},\mathit{b}}={\displaystyle \sum _{\mathit{v},\mathit{h}}}E\left(\mathit{v},\mathit{h}\right)-{\displaystyle \sum _{\mathit{h}}}E\left(\mathit{v},\mathit{h}\right)$$

(4)

Normally Equation (4) can also be written as:

$$\frac{\partial \log \left(P\left(\mathit{v}\right)\right)}{\partial \mathit{W},\mathit{a},\mathit{b}}=<v_i{h}_j{>}_{data}-<v_i{h}_j{>}_{model}$$

(5)

where < > denotes the expectation. However, the expectation $<v_i{h}_j{>}_{model}$ in the maximum log likelihood function cannot be easily computed, and is thus estimated using contrastive divergence which leads to the following parameter updating equation [30]:

$$\begin{array}{c}\begin{array}{cc}\hfill {W_{ij}}^k& ={W_{ij}}^k+\gamma \left({<{v_i}^k{h}_j>}_{data}-<{v_i}^k{h}_j{>}_T\right)\hfill \\ & ={W_{ij}}^k+\gamma \left[P(h_j=1| \mathit{v})v_i-{v_i}^k P(h_j=1| {\mathit{v}}^k)\right]\hfill \end{array}\\ \begin{array}{cc}\hfill {a_i}^k& ={a_i}^k+\gamma \left({<{v_i}^k>}_{data}-<{v_i}^k{>}_T\right)\hfill \\ & ={a_i}^k+\gamma \left[P(v_i=1| \mathit{h})-{v_i}^k\right]\hfill \end{array}\\ \begin{array}{cc}\hfill {b_j}^k& ={b_j}^k+\gamma \left({<h_j>}_{data}-{<{h_j}^k>}_T\right)\hfill \\ & ={b_j}^k+\gamma \left[P(h_j=1| \mathit{v})-P(h_j=1| {\mathit{v}}^k)\right]\hfill \end{array}\end{array}$$

(6)

where $T$ represents a full step of Gibbs sampling, $\gamma$ represents the learning rate and $k$ represents the $k-\mathrm{step}$ of contrastive divergence. The neuron activation probabilities are given by the following equations:

$$P(h_j=1| \mathit{v})=\sigma (b_j+{\displaystyle \sum _{i=1}^n}{W}_{ij}{v}_i)$$

(7)

$$P(v_i=1| \mathit{h})=\sigma (a_i+{\displaystyle \sum _{j=1}^m}{W}_{ij}{h}_j)$$

(8)

where $n$ represents the number of visible units, $m$ the number of hidden units, and σ is the activation function. The activation function is typically the logistic function used as a threshold defined as:

$$\sigma \left(x\right)=\frac{1}{1+e^{-x}}$$

(9)

2.2. The Deep Belief Network

A DBN is formed by stacking multiple RBMs on top of each other (see Figure 2) in order to create high representations of data that can be used for classification, regression (continuous output) tasks as well as unsupervised learning.

The RBM becomes a building block for forming a deep belief network. The DBN can be trained in a greedy-layer-wise fashion by stacking RBMs on top of each other [31]. The output of one RBM, that is the activation values in the hidden layer, simply becomes the input for the next RBM, and the parameters of the previous RBM do not change. This next RBM is trained by the same process as illustrated in the previous section. This process allows for creating multiple hidden layers.

In order to use a DBN to predict a continuous output, one can first learn the weights and biases in the unsupervised stage of learning. Once the optimal parameters (weights and biases) have been determined, a supervised fine tuning stage is performed. This is done by creating a final output layer on the top of the DBN which outputs the predicted RUL value, given a set of vibration features. This is illustrated in Figure 3. The parameters of the entire network are then updated using the back-propagation algorithm in the same way as a feedforward neural network (FNN) is trained. In this way, the DBN pre-trains the network, which serves as an initialization step for the parameters of the FNN, instead of a random initialization of the weights and biases, which has been shown to add robustness to deep architectures and decrease the probability of obtaining a poor local minima [32].

The DBN performs unsupervised learning first by training it on a set of signal features in order to learn a latent representation of the data. After the end of training, an output layer is added on top of the last layer of the DBN, where it is fully connected to the (last) hidden layer. This output layer contains one neuron (without an activation function) which represents the continuous prediction.

2.3. Particle Filter

The particle filter is a Monte Carlo approach that can be used to estimate the state of a system by combining both the state evolution of the system and the observation/measurement parameters obtained from the state.

In discrete time, a system can be described by the following state space model:

$${\mathit{x}}_t=f\left({\mathit{x}}_{t-1},{\mathit{\omega }}_{t-1}\right)$$

(10)

$${\mathit{z}}_t=h\left({\mathit{x}}_t,{\mathit{v}}_t\right)$$

(11)

From the above equations, ${\mathit{x}}_t$ represents the state of the system (such as the crack depth) at time $t$, ${\mathit{\omega }}_{t-1}$ which is independently and contains identically distributed (iid) noise/variance at time $t-1,$ $f( )$ is a function that maps the transitions between states, ${\mathit{z}}_t$ represents the measurement/observation (can be thought of as a feature, typically derived from a vibration signal) at time $t$, ${\mathit{v}}_t$ is the iid noise/variance associated with each measurement, and $h( )$ is a function that maps the state with the measurement.

The goal of the particle filter is then to be able to estimate the probability density function (pdf) of $p({\mathit{x}}_t|{\mathit{z}}_{1:t})$, that is, to estimate the state at time $t$, given the measurements up to time $t.$ It is also assumed that the prior probability of the state $P\left({\mathit{x}}_0\right)$ is unkown.

In the Bayesian setting, the state estimation is usually computed recursively in two stages; the prediction and the update. For the prediction step, the $P({\mathit{x}}_t|{\mathit{z}}_{1:t-1})$ is given as:

$$\begin{array}{c}\int P\left({\mathit{x}}_t|{\mathit{x}}_{t-1},{\mathit{z}}_{1:t-1}\right)P\left({\mathit{x}}_{t-1}|{\mathit{z}}_{1:t-1}\right)d{\mathit{x}}_{t-1}\\ =\int P\left({\mathit{x}}_t|{\mathit{x}}_{t-1}\right)P\left({\mathit{x}}_{t-1}|{\mathit{z}}_{1:t-1}\right)d{\mathit{x}}_{t-1}\end{array}$$

(12)

For the update step, new measurements are collected, which are then used to update the prior distribution. The posterior distribution can then be written as:

$$P\left({\mathit{x}}_t|{\mathit{z}}_{1:t}\right)=\frac{P\left({\mathit{x}}_t|{\mathit{z}}_{1:t-1}\right)P\left({\mathit{z}}_t|{\mathit{x}}_t\right)}{\int P\left({\mathit{x}}_t|{\mathit{z}}_{1:t-1}\right)P\left({\mathit{z}}_t|{\mathit{x}}_t\right)d{\mathit{x}}_{t-1}}$$

(13)

Typically, the calculations for Equations (12) and (13) are intractable and the particle filter sampling approach is used to approximate them. This can be accomplished by using a set of samples/particles ${\{{{\mathit{x}}_t}^i, {w_t}^i\}}_{i=1}^N$, where ${w_t}^i$ is the ith weight at time $t$ and $N$ is the user-specified number of particles generated. The weights can be updated by using the sequential importance resampling (SIR) algorithm [33], and samples are drawn from the state transition distribution:

$${\mathit{x}}_t^i~p({\mathit{x}}_t|{\mathit{x}}_{t-1}^i)$$

(14)

$$w_t^i=\frac{p({\mathit{z}}_t|{\mathit{x}}_t^i)}{{\sum }_{i=1}^{N}p({\mathit{z}}_t|{\mathit{x}}_t^i)}$$

(15)

Finally, the posterior distribution $P({\mathit{x}}_t|{\mathit{z}}_{1:t-1})$ is obtained by resampling from $\left\{{\mathit{x}}_t^1,{\mathit{x}}_t^2,\dots {\mathit{x}}_t^N\right\}$ where $\widehat{{\mathit{x}}_t^i}$ (predicted estimate) is drawn with probability $w_t^i$.

2.4. Combined DBN and Particle Filter-Based Approaches for RUL Prediction

The ultimate objective of the combined DBN and particle filter approach is to estimate the pdf of $P\left({\mathit{x}}_{t+L}|{\mathit{z}}_{1:t}\right)$, where $L ϵ \{1,2,\dots N-t\}$ and $N$ represents the length of the signal. It should be noted that no new (future) information about the system is provided at the generic time step $t$, and only information provided from time steps $\{0,1,\dots t\}$ may be used for the prediction.

There are two functions that are of interest in modeling; the state transition distribution Equation (14) (sometimes referred to as the proposal distribution) and the measurement distribution Equation (15). Both of these functions can be modeled by the DBN, since neural networks in general are universal function approximators [34].

In the prediction step, the state transition model can simply be approximated by first reconstructing the time series of the measurements (${\mathit{z}}_t$ for $t=1,2,\dots N)$ into a matrix, where each feature (column) represents a lagged order of the time series, and whose output is an L-step ahead into the future state value, and each row represents an index in time. Formally, the input can be denoted as:

$$\left[{\mathit{z}}_t,{\mathit{z}}_{t-1},\dots .,{\mathit{z}}_{t-d+1}\right], ϵ {ℝ}^d$$

(16)

and treat the state ${\mathit{x}}_t$ as the actual RUL of the bearing at time $t$ as the output:

$$\left[x_t,x_{t+1},\dots ,x_N\right]$$

(17)

where $d$ represents the embedding dimension, and determines the size of the first visible layer in the DBN. Thus, the network requires training tuples of the form $\left(\left[{\mathit{z}}_t, {\mathit{z}}_{t-1}, \dots , {\mathit{z}}_{t-d+1}\right], x_t\right)$, $t=d-1,\dots ,N$. Using this training set, the state transition model is instead actually modeling the pdf of $p({\mathit{x}}_{t+L}^i|{\mathit{z}}_t,{\mathit{z}}_{t-1},\dots .,{\mathit{z}}_{t-d+1})$. The $L$ step ahead prediction, $x_{t+L}^i,$ is simply made by subtracting the network’s output $x_t$ by $L$. The advantages of treating the state $x_t$ (instead of treating it as the crack depth for instance) as the actual RUL allow for a simpler model to be developed. One does not need to utilize another neural network to determine the RUL, given the crack depth, or to recursively compute the state transition model multiple times until it exceeds such a threshold, which needs to be defined. By training the network directly on the RUL, the network is minimizing the error of predicting the RUL, rather than an intermediate value. This results in a network that is more easily trainable and reduces the potential problem of having the network’s eventual forecast of the state die off (converging to a single value). The disadvantage of this approach however, is that the parameters of the state transition model will be updated/relearned less frequently. The measurement ${\mathit{z}}_t$ (often a multi dimensional vector) can be constructed into a mono-dimensional feature vector by setting the input of the DBN as ${\mathit{z}}_t$ and then setting the size of the last hidden layer to one. The output of the DBN in its unsupervised stage of learning is then the reconstructed mono-dimensional feature vector of ${\mathit{z}}_t$.

Equations (16) and (17) define a method often called the sliding window technique [35]. This windowing approach is illustrated in Figure 4.

The first plot in Figure 4 is the complete series of a fault feature. The second plot highlights the first window of size $d$ = 100. The last figure simply shows a zoom-in of the first window. The first window contains points from $t=0, 1, 2, \dots 98, 99$ (first row of features), and the second window would contain points from $t=1, 2, 3,\dots 99, 100$ (second row of features), and so on.

The state transition model and its respective variance in Equation (10) ${\mathit{\omega }}_{t-1}$ can be modeled by taking $B$ bootstraps, which randomly samples (with replacement) each row of the training data $TD,$ $\left|TD\right|$ times. The DBN is then trained $B$ times for each bootstrap and the predicted values will be defined as:

$$S{T}_b\left(\mathit{z}\right)=F\left(\mathit{z};S{T}_b^{*}\right)$$

(18)

where, $\mathit{z}$ represents the generic testing input vector of the past $d$ measurements [Equation (16)] and $F\left(\mathit{z};S{T}_b^{*}\right)$ represents the DBN-FNN output [Equation (17)] with respect to the $b\mathrm{th}$ bootstrap. The mean and variance are simply then calculated as:

$$S{T}_{avg}\left(\mathit{z}\right)=\frac{1}{B}{\displaystyle \sum _{b=1}^B}S{T}_b\left(\mathit{z}\right)$$

(19)

$${\widehat{\sigma }}_{ST}^2\left(\mathit{z}\right)=var\left(S{T}_b\left(\mathit{z}\right)\right)$$

(20)

and then samples $x_t^i$ can be generated from the following Gaussian distribution:

$$\mathsf{Ɲ}(S{T}_{avg}(\mathit{z}),{\hat{\sigma }}_{ST}^2(\mathit{z}))$$

(21)

The measurement distribution can be modeled in a somewhat similar approach to the state transition distribution. The technique [13] requires a dataset of tuples of ${\left(x_t,{\mathit{z}}_t\right)}_{t=1}^{N_{training}}$ where $B$ bootstraps are sampled from this training data set, and an interpolator $\phi \left(x\right)$ is created. ${\mathit{z}}_t$ is typically a multi-dimensional vector, which can be reduced to a single mono-dimensional vector for simplicity and ease of calculation. This can be performed by the DBN in its unsupervised stage of learning, by setting the last layer of the DBN’s hidden layer to a size of one. A simple FFN can also be used as an interpolator for computing the following:

$${\phi }_{avg}\left(x\right)= \frac{1}{B}{\displaystyle \sum _{b=1}^B}\phi \left(x;T_b^{*}\right)$$

(22)

where, $T_b^{*}$ is the $b$th bootstrap sampled from the dataset. The measurement model in Equation (11) can be hypothesized as:

$$z\left(x\right)=f\left(x\right)+v\left(x\right)$$

(23)

From Equation (23), we can subtract ${\phi }_{avg}\left(x\right)$ from both sides such that:

$$z\left(x\right)-{\phi }_{avg}\left(x\right) =\left[f\left(x\right)-z\left(x\right)\right]+v\left(x\right)$$

(24)

The error term $z\left(x\right)-{\phi }_{avg}\left(x\right)$ is then a function of $\left[f\left(x\right)-z\left(x\right)\right]$ and $v\left(x\right)$, which we will call the model error and intrinsic noise respectively. Their variances are then denoted as ${\sigma }_m^2\left(x\right)$ and ${\alpha }^2\left(x\right)$ respectively. The model error variance can be modeled by taking $M$ groups of interpolators from $\phi \left(x;T_b^{*}\right)$ of length $K$ networks and computing the average as:

$${\phi }_{com}^m\left(x\right)=\frac{1}{K}{\displaystyle \sum _{i=1}^K}{\phi }_i\left(x\right)$$

(25)

The set, $\Psi ={\{{\phi }_{com}^m\left(x\right)\}}_{m=1}^M$ is then resampled with replacement using $P$ bootstraps, denoted as ${\{{\Psi }_p\}}_{p=1}^P$ , where ${\Psi }_p$ is the $p$th bootstrap of $\Psi$. The estimate of the model error variance can then be computed as:

$${\hat{\sigma }}_m^2\left(x\right)=\frac{1}{P}{\displaystyle \sum _{p=1}^P}var({\Psi }_p)$$

(26)

The noise variance can be modeled by building up a training set ${(x_t,{\hat{\alpha }}_t^2)}_{t=1}^{N_{training}}$, where ${\hat{\alpha }}_t^2$ is defined as:

$${\hat{\alpha }}_t^2=max\left\{{\left({\mathit{z}}_t-{\phi }_{avg}\left(x_t\right)\right)}^2-{\hat{\sigma }}_m^2\left(x_t\right),0\right\}$$

(27)

A single FNN can then be trained on the aforementioned training set in order to estimate ${\hat{\alpha }}_t^2\left(x\right)$ for an arbitrary $x$ vector. Finally $p({\mathit{z}}_t|x_t)$ can be approximated by:

$$p\left({\mathit{z}}_t|x_t\right)\approx \mathsf{Ɲ}\left({\phi }_{avg}\left(x\right),{\hat{\sigma }}_m^2\left(x\right)+{\hat{\alpha }}_t^2\left(x\right)\right)$$

(28)

The confidence intervals for a generic prediction $\hat{x}$ can then be obtained with 1-$\alpha$% confidence by the following formula [36]:

$$\hat{x}\pm t_{1-\left(\alpha /2\right),B-1}\sqrt{\frac{{\hat{\sigma }}_{ST}^2\left(x\right)}{B-1}}$$

(29)

The training/testing procedure for the integrated DBN and particle filter is then as follows:

Step 1	Let the last 100 rows be equal to the testing set, and set the rest of the data to the training set.
Step 2	Approximate the state transition model.
	a	Reconstruct the time series of the signal features into a matrix with an embedding dimension of $d$. This will serve as the input, and the output will be the mapped state $x_t$, where $t=d-1,\dots ,N_{training}-L-1$.
	b	Create $B$ bootstrapped data sets using the data created in Step 1.
	c	Initialize the weights and biases of the FNN by training the DBN on the input data with all the data except for the last 100 rows. The rest of the data is the testing set.
	d	Train the FNN. Fine tune the weights in a supervised fashion by minimizing the loss function on the training set and by using the back-propagation algorithm.
	e	Predict the $x_{t+L}$. This is accomplished by subtracting the network’s output $x_t$ by L, i.e., $x_{t+L}$ = $x_t-L$.
	f	Let $t=t+1$, and update the training vector with input features $\left({\mathit{z}}_t,{\mathit{z}}_{t-1},\dots .,{\mathit{z}}_{t-d+1}\right)$.
	g	Repeat Steps 2e–2f, until all 100 points have been predicted.
Step 3	Estimate the measurement distribution. Create a new dataset of the form ${\left(x_t,{\mathit{z}}_t\right)}_{t=1}^{N_{training}}$ Generate B bootstraps from this training dataset and apply Equations (22)–(28) to obtain the measurement distribution. The DBN can be used for estimates in Equations (22) and (27).
Step 4	Estimate the state. The predicted state ${\widehat{x}}_{t+L}$ can be obtained by sampling $\{x_{t+L}^1,x_{t+L}^2,\dots x_{t+L}^B\}$ with probability $w_t^i$ [from Equation (15)].

This method allows a complete data-driven approach to be developed in conjunction with a particle filter for RUL estimation. Equation (10), the proposal distribution is built up using training tuples of the form $\left(\left[{\mathit{z}}_t,{\mathit{z}}_{t-1},\dots .,{\mathit{z}}_{t-d+1}\right],x_t\right)$. The bootstrapping approach is then used to collect the statistics for the mean and variance. Equation (11), the measurement model is then modeled by Equations (22)–(28). Here the DBN can serve two purposes. It can be used to reduce the dimensionality of the measurement ${\mathit{z}}_t$ and it can also to be used in order to initialize the weights of the neural network.

3. Validation

In this section, the validation of the integrated bearing RUL prediction method using vibration data collected from a single hybrid ceramic bearing run-to-failure test is discussed.

3.1. Hybrid Ceramic Bearing Run-to-Failure Test Setup

The hybrid ceramic bearing run-to-failure tests were performed using a bearing test rig in the laboratory as shown in Figure 5. The bearing test rig consisted of the following main components: (1) 3-HP AC motor with a maximum speed up to 3600 rpm and variable speed controller; (2) Hydraulic dynamic loading system with a maximum radial load up to 4400 lbs or 19.64 kN; (3) Integrated loading and bearing housing. The rig can be used for testing both ball and tapered roller bearings.

An automatic data acquisition system was constructed using a National Instrument CI 4462 board (NI, Austin, TX, USA) and NI LabVIEW software (LabView 2012, NI, Austin, TX, USA). The automatic data acquisition system is characterized with the following key features: (1) Maximum sampling rate up to 102.4 kHz; (2) 4 Input simultaneous anti-aliasing filters; (3) Software-configurable AC/DC coupling and IEPE conditioning; (4) Vibration analysis functions such as envelope analysis, cepstrum analysis, and so on for computing necessary condition indicators.

The tested hybrid ceramic ball bearing was a SMR6205C-ZZ/C3 #3 L55/MG2 type bearing by Boca Bearing Company (Boca Bearings, Boynton Beach, FL, USA). It consisted of stainless steel inner outer races, and ceramic balls. The bearing was mounted on the test rig. Two accelerometers were stunt mounted on the bearing housing in the direction perpendicular to the shaft. During the tests, the rig was run at a speed of 1800 rpm (30 Hz) and was subjected to a radial load of 600 psi. Vibration data were collected with a sampling rate of 102.4 kHz for two seconds at each sampling point. There was a 5 min gap between any two sampling points. At the end of the test, the test bearing was disassembled, checked, and photographed. The bearing data contained a total of 849 data files with a length of approximately 71 hours.

Table 1. The settings of the run-to-failure test.

Test Bearing Name	Type of Bearing	Load (psi)	Input Shaft Speed (hz)
B2	Hybrid ceramic bearing	600	30

describes the run-to-failure test setting.

Table 2. Specifications of the test bearing.

Parameter	Specification
Bearing material	Stainless steel 440c
Ball material	Ceramic SI3N4
Inner diameter (d)	25 mm
Outer diameter (D)	52 mm
Width	15 mm
Enclosure	Two shields
Enclosure material	Stainless steel
Enclosure type	Removable (S)
Retainer material	Stainless steel
ABEC/ISO rating	ABEC #3/ISOP6
Radial play	C3
Lube	Klubber L55 grease
RPM grease (×1000 rpm)	19
RPM oil (×1000):	22
Dynamic load (kgf)	1429
Basic load (kgf)	804
Working temperature (°C)	121
Weight (g)	110.32

provides the specifications of the tested bearing.

The root mean square (RMS) of the vibration signals was computed to represent the degradation of the bearing over time during the run to failure tests. The RMS at each time interval (denoted as $RM{S}_t$) can be calculated as follows:

$$RM{S}_t=\sqrt{\frac{1}{n}{\displaystyle \sum _{i=1}^n}{f}_{ti}^2}$$

(30)

where $f_{ti}$ represents the $i\mathrm{th}$ raw vibration data point at time interval $t$ and $n$ is the length of the signal at time interval $t$. The RMS for the bearing data can be seen in Figure 6.

For the bearing data, the $RU{L}_t$ (remaining useful life at time $t$) was calculated simply by taking the time index of the maximum recorded RMS value as the point of failure denoted as $RM{S}_{T_{end}}$ and subtracting it from each time step:

$$RU{L}_t=RM{S}_{T_{end}}-t$$

(31)

The raw vibration signals were preprocessed using the fast Fourier transform (FFT), and the FFT values were used as the only signal feature input into the integrated DBN-particle filter model to predict the RUL of the bearing. The FFT, which is an efficient algorithm for computing the discrete Fourier transform (DFT) at a time interval $t$, can be calculated as follows:

$$DF{T}_{tn}={\displaystyle \sum _{k=0}^{N-1}}{f}_{tk}{e}^{\frac{-2\pi ikn}{N}}$$

(32)

where $f_{tk}$ is the $k\mathrm{th}$ raw vibration signal at time interval $t$, $N$ is the length of the signal at time interval $t$, $i=\sqrt{-1}$, and $n$ = 0,1,…,$N-1$.

Equation (31) transforms the vibration signals from a time domain to a frequency domain in which we extracted ten equal bands ranging from 0 to 20 kHz.

3.2. Hybrid Ceramic Bearing RUL Prediction Results

A total of 849 time steps were extracted from the bearing data in which all the data up until time step 792 were used. As seen from Figure 6, a rather large dip in the RMS values occurred from time steps 720 through 735. Features collected from those points were simply removed from the data and treated as outliers.

$L$ = 1 and $L=10$ were used to predict 5 min and 50 min into the future for the bearing data, respectively.

The predicted RUL values for the last 100 steps can be seen in Figure 7 and Figure 8 for $L$ = 1 and $L$ = 10, respectively. The error metrics and hyperparameters of the DBN with $L$ = 1 and $L$ = 10 are provided in

Table 3. Root mean squared error (RMSE) and mean absolute percentage error (MAPE) results.

Combined DBN and Particle Filter-Based Approach
$\mathit{L}$	RMSE	MAPE	$\mathsf{\alpha }-\mathsf{\lambda }$ Accuracy (α = 10%)
1	2.04	7.33%	0.80
10	3.52	8.68%	0.61
Particle Filter-Based Approach
$\mathit{L}$	RMSE	MAPE	$\mathsf{\alpha }-\mathsf{\lambda }$ Accuracy (α = 10%)
1	2.53	7.47%	0.71
10	3.65	8.73%	0.53

and

Table 4. Hyperparameters of the deep belief network (DBN).

$\mathit{L}$	DBN Learning Rate	DBN Epochs	Hidden Layer Structure	FNN Learning Rate	FNN Epochs
1	0.002	74	[146, 53]	0.0017	176
10	0.0023	82	[120, 54]	0.0014	92

, respectively. Table 4 shows the hyperparameters of the DBN for the state transition model which were determined using a grid search. Input data was also scaled to be in [0, 1], $d$ = 100 was set as the embedding dimension, $B$ was set as 50, $M=10, K=5, P=1000$ and 50 particles were used for both $L$ = 1 and $L$ = 10 predictions.

In Figure 7 and Figure 8, the green color represents the average predicted RUL values across the bootstrapped samples. The red error bars represent the 95% bounds. The predicted results show for both $L$ = 1 and $L$ = 10 that it can accurately predict the true RUL, and as the bearing approaches the point of failure, the accuracy of the predictions tends to increase.

The confidence bounds for the $L$ = 10 predicts the RUL of the bearing slightly early when compared to the $L$ = 1 predictions and exhibits a greater variance.

The common error metrics used in Table 3 are the root mean squared error (RMSE), $\mathsf{\alpha }-\mathsf{\lambda }$ metric [37,38], and the mean absolute percentage error (MAPE). The MAPE, RMSE, and $\mathsf{\alpha }-\mathsf{\lambda }$ metric are defined by the following equations:

$$MAPE=\left(\frac{1}{n}{\displaystyle \sum _{t=1}^n}\frac{A_t-F_t}{A_t}\right)\ast 100\%$$

(33)

$$RMSE =\sqrt{\frac{1}{n}{\displaystyle \sum _{t=1}^n}{(A_t-F_t)}^2}$$

(34)

$${\left(\mathsf{\alpha }-\mathsf{\lambda }\right)}_{t_{\mathsf{\lambda }}}=\{\begin{array}{cc}1& if \left(1-\mathsf{\alpha }\right)RUL{ }_{t_{\mathsf{\lambda }}}\le \widehat{RUL{ }_{t_{\mathsf{\lambda }}}}\le \left(1+\mathsf{\alpha }\right)RUL{ }_{t_{\mathsf{\lambda }}}\\ & 0 otherwise\end{array}$$

(35)

In Equations (33) and (34), $A_t$ denotes the actual value, $F_t$ represents the predicted value, and $n$ equals the number of points to be predicted. In Equation (35), $\mathsf{\alpha }$ represents a user specified bound level, $RUL{ }_{t_{\mathsf{\lambda }}}$ represents the actual RUL at time $t_{\mathsf{\lambda }}$, and $\widehat{RUL{ }_{t_{\mathsf{\lambda }}}}$ represents the predicted RUL at time $t_{\mathsf{\lambda }}$. $\mathsf{\lambda }$ is given as a percentage of RUL for a given equipment (i.e., $t_{\mathsf{\lambda }=50\%}$ represents the time index where half the RUL is left). The reported $\mathsf{\alpha }-\mathsf{\lambda }$ in Table 3 represents the average $\mathsf{\alpha }-\mathsf{\lambda }$ metric across the all testing samples. In the $\mathsf{\alpha }-\mathsf{\lambda }$ metric, results closer to one indicate better performance. The loss functions used to train the state transition models for $L=1$ and $L=10$ were the MSE ($RMSE{ }^2$) and the MAPE respectively. Satisfactory hyperparameters for building up Equations (22) and (27) for both $L=1$ and $L=10$ were set as [105, 59] for the hidden layer structure using 122 epochs with a learning rate of 0.00085. The hyperparameters for the reconstructed mono-dimensional measurement data were set as [142, 87] for the hidden layer structure using a small learning rate of 0.0094 and 212 epochs. The hyperparameters for each built network was done by employing a grid search and evaluating candidate hyperparameters on the MSE using a 10 fold cross validation on the training set. Cross validation is a widely used method to avoid overftting when selecting hyperparameters [39]. The chosen hyperparameters were obtained by those that minimized the MSE during cross validation. For all of these networks, the activation function was set as the rectified linear unit (ReLU) function. It solves the vanishing gradient problem that other non-linear activation functions can cause [40]. The ReLU activation function for the input $x$ of a neuron is defined as:

$$ReLU(x)=max\{0,x\}$$

(36)

Since the particle filter is the most competitive RUL prediction method for bearings, for a comparison purpose, the RMSE and MAPE of the RUL predictions obtained by the particle filter-based approach are also provided in Table 3. In comparison with the results obtained using the particle filter-based approach, the RMSE and MAPE values of the integrated approach were slightly lower and the $\mathsf{\alpha }-\mathsf{\lambda }$ metric values were higher. Note that since in [28], it was reported that using the same bearing data, the particle filter gave a better RUL prediction accuracy than a DBN based approach did, we consider the integrated method presented here as better than the DBN-based approach used on the same dataset. The comparison results showed the promising performance of combining the deep learning based approach with particle filter for hybrid ceramic bearing RUL prediction.

Given that the integrated approach did not require explicit model equations like the particle filter-based approach and is scalable for big data applications, the RUL prediction performance achieved by the integrated approach has shown great potential for bearing RUL prediction with big data.

4. Conclusions

Predicting the remaining useful life of bearings has been an important task for condition-based maintenance of industrial machines, as bearings are one of the most critical components in these machines. To meet the challenge of automatically processing massive data and accurately predicting RUL of the bearings in the era of the Internet of Things and Industrial 4.0, this paper addressed the limitations of traditional data-driven prognostics by presenting a new method that integrates a deep belief network and a particle filter for RUL prediction of hybrid ceramic bearings. Real vibration data collected from hybrid ceramic bearing run-to-failure tests were used to test and validate the integrated method. The performance of the integrated method was also compared with DBN and particle filter-based approaches. In comparison with the RUL prediction results obtained using the particle filter-based method, the prediction accuracy measured by RMSE and MAPE of the integrated method is slightly lower. Also, based on the previous comparison between DBN and the particle filter methods for the same dataset reported in [28], the RUL prediction performance of the integrated method is considered better than the DBN method. The validation and comparison results showed the promising RUL prediction performance of the integrated method.

Since the integrated approach was a purely data-driven approach, future work should focus on increasing user confidence in the method. This may be accomplished by combining the integrated approach with other techniques such as employing the use of fuzzy similarity [41]. This would require a large database of run to fail trajectories that can be compared based on their similarity with the observed data. This large database should not only increase the accuracy of the integrated approach, but also further validate it. Additional work may also include the processing of the raw vibration signals at each sampling interval by a DBN into a single dimension signal feature that can be used for prognostics.