Application of Machine Learning Tools for Long-Term Diagnostic Feature Data Segmentation

Abstract

In this paper, a novel method for long-term data segmentation in the context of machine health prognosis is presented. The purpose of the method is to find borders between three data segments. It is assumed that each segment contains the data that represent different statistical properties, that is, a different model. It is proposed to use a moving window approach, statistical parametrization of the data in the window, and simple clustering techniques. Moreover, it is found that features are highly correlated, so principal component analysis is exploited. We find that the probability density function of the first principal component may be sufficient to find borders between classes. We consider two cases of data distributions, Gaussian and $\alpha$-stable, belonging to the class of non-Gaussian heavy-tailed distributions. It is shown that for random components with Gaussian distribution, the proposed methodology is very effective, while for the non-Gaussian case, both features and the concept of moving window should be re-considered. Finally, the procedure is tested for real data sets. The results provided here may be helpful in understanding some specific cases of machine health prognosis in the presence of non-Gaussian noise. The proposed approach is model free, and thus it is universal. The methodology can be applied for any long-term data where segmentation is crucial for the data processing.

1. Introduction

Condition monitoring systems are designed to collect a massive amount of data during the operation of machines. The purpose of this is to assess the condition of the machine. Typically, there is a set of features extracted from raw signals, and each of features is compared to so-called limit values (threshold corresponding to change from “Good Condition” to “Warning” and “Warning” to “Alarm”). In some cases, such limit values are provided by the manufacturer. Unfortunately, in many cases, especially when the machine is pretty unique, we do not know about the limit values or expected lifetime.

In the paper, we propose an approach that based on historical data may identify the border of classes good condition/warning/alarm, i.e., identify the points in the data when the machine changes the condition. The change in condition is manifested by the change in statistical properties of the collected data. There are many models of the lifetime curve. Here, we follow the shape of the curve that consists of three regimes. In the first regime, there is a nearly constant value of feature with some small variation. Next, in regime two, there is linear growth also, and the variation of random components is smoothly growth. Finally, in regime three, the feature is exponentially increasing, and the random component has significant variation. Figure 1 presents the conceptual model of the input data.

The data describing the degradation of the machine are not purely deterministic and they contain some randomness. In this paper, we assume two cases. In the first one, the random component follows the Gaussian distribution and, as mentioned, its variance is constant in the first segment, linearly increasing over time in the second segment and finally exponentially increasing in the third segment.

In the second case, the character of the noise is non-Gaussian, and we assume it can be modeled by $\alpha$-stable distribution with $\alpha <2$. Moreover, the scale parameter $\sigma$ is changing over time (it corresponds to variance for Gaussian distribution).

As the raw data are a random, non-stationary process, we propose to use moving window with fixed size for primary segmentation, then we describe the data in each segment by simple statistical descriptors (sample mean, sample variance, sample kurtosis, etc.). Obviously, such features are highly redundant, especially when there is many of them; thus the basic technique for dimensionality reduction is applied. Finally, the reduced data set is clustered by various machine learning algorithms, and the most efficient scenario is selected. Such an approach is quite simple and effective for the Gaussian distribution case. Unfortunately, for the non-Gaussian case, both statistical parameters as well as clustering techniques must be deeply revised, as $\alpha$-stable distributed data contain many outliers. In the paper, we propose a methodology for finding the limit values for the mentioned classes (good condition/warning/alarm), and we illustrate the efficiency of approach for simulated and real data. We explain that the source of relatively poor efficiency may be related to a non-Gaussian distribution of random components, and we highlight that robust methods for parameterization and clustering are needed.

2. State of the Art

There is a lot of data collected every minute in various branches of the economy, such as industry, financial sector and many others. Data regarding currency exchange [1], stock exchange, raw materials prices [2,3], energy consumption/prices [4,5], etc., are collected, processed and analyzed to understand the nature of important phenomena. However, each case is specific and requires appropriate methodology for data analysis. In this paper, we focus on long-term data from condition monitoring (CM) systems; however, some inspirations from other areas are also helpful.

Nowadays, the condition monitoring systems are frequently used to monitor current state of the given machine. Although there are at least several freely-available data sets, there are just a few practical examples of how to process and analyze long-term data acquired in industrial installations. There are many papers on data modeling for prognosis, but these works seem to be more theoretical (data sets from test rigs, accelerated fatigue tests with artificially initiated fault, etc.) than really applied to the actual industrial systems with natural degradation processes.

Just to mention several practical implementations, one may focus on [6] where the procedure of load-dependent features processing with application to wind turbine bearings was proposed. Another example proposed by Ignasiak et al. [7] discussed a comparative study on statistical and energy-based parameters analysis in a long-term context, also for a wind turbine. Wodecki et al. [8] analyzed the overheating problem for heavy-duty machines operating in the underground mine. Their method focused on temperature change detection in the long-term data set using Anderson–Darling statistics. Grzesiek et al. [9] proposed a statistical test for anomaly detection in long-term temperature data analysis for gearbox diagnostics used in underground mine. Wang et al. [10] introduced a hybrid prognostics method for estimating the remaining useful life of wind turbine bearings. Li et al. [11] developed an approach for predicting for a lithium-ion battery under vibration stress using the Elman neural network.

Analysis of the non-stationary data using moving window is commonly used to investigate local properties of the data, and was used, among others, by Staszewski et al. (moving window and kurtosis estimation in raw vibration for local damage detection) [12] or Zimroz et al. (moving window for HI from wind turbine condition monitoring system to estimate regression between health index (HI) and operational parameters) [6].

It was also shown that window selection is critical in forecasting [13]. In signal processing, the window length selection was clearly discussed in [14], where short time Fourier transform (STFT) was proposed to analyze non-stationary data. Haoran Yan et al. [15] introduced a methodology based on long short-term memory for predicting gear remaining useful life (RUL). As mentioned, simple statistics such as sample mean, sample variance, sample kurtosis, etc., are often used in condition monitoring for signal parameterization. In this paper, it is proposed to use such statistical descriptors in order to differentiate segments from different regimes. Initially, several statistics with theoretical potential in considered context are proposed. Finally, the mean value, standard deviation (STD) and root mean square (RMS) are applied for each segment. Machine learning is frequently used for long-term data analysis for classification, detection, prognosis, etc., [16,17,18,19,20,21,22,23,24,25,26,27]. In this case, unsupervised classification, i.e., clustering, is required because every machine works in different conditions, and labeling data is not an easy task. Hence, the clustering approach has massive potential for dividing long-term data into several regimes. The authors of [17] proposed the long short-term memory (LSTM) network with the clustering method for multi-stage predicting RUL. Jaskaran Singh et al. [18] introduced an adaptive data-driven model-based approach to detecting regime-changing point by using K-means clustering. Additionally, there are a lot of applications for regime changing point detection in time series in different areas, such as financial data analysis, etc. [28,29,30,31,32,33,34,35,36,37,38]. Detection of change in time series is a very widely investigated problem.

Signal segmentation is frequently used in various signal processing applications to divide original data into homogeneous segments or to extract the pattern, etc. Kucharczyk et al. [39] used stochastic modeling for seismic signal segmentation. Gasior et al. [40] used segmentation for shock extraction in a sieving screen vibrations. A challenge in signal segmentation is when the difference between regimes is not so clear. One may say that regime A is smoothly transforming to regime B. This issue was investigated by Grzesiek et al. [41]. It is also the case here, especially as there is “no jump” from good condition to warning state, that the machine in good condition slowly starts the degradation process. Another issue is multiple change points and unit heterogeneity [42].

3. Problem Formulation

The prognosis in predictive maintenance is related to the forecasting of HI values for future time instances or estimation of a RUL time [43,44,45,46,47,48]. In any case, to deal with prediction, one needs a model of the degradation of the machine. It is the most frequent case that advanced mathematical models or machine learning (ML) approaches are used for prognosis. ML approach requires historical data to train the prediction system or establish a data-driven model for the prognosis. From the reliability theory, one may notice that there are several “general” degradation models. As mentioned, in this paper, we follow the idea that machine lifetime consists of three phases (see Figure 1):

(a)

Good condition, where HI is nearly constant (no degradation);

(b)

Slow degradation (HI is increasing slowly and approximately linearly);

(c)

Fast degradation (HI is rapidly growing like exponential function).

In such a case, the model of the degradation, in fact, consists of three sub-models (three regimes). It means that before the identification of the model, the data need to be segmented. Segments corresponding to given sub-models should be extracted, and the data in each segment should be modeled separately. Thus, for long-term data from the condition monitoring system, one needs to apply the segmentation procedure before modeling. In other words, one needs to find so called regime change points, where the data are changing their nature in the sense of statistical properties. In this paper, the general idea is to analyze locally statistical parameters by the moving window approach. For a priori selected window length and overlapping, simple descriptive statistics are calculated, and then these features for each segment are subjected to clustering. There are some challenging issues to solve. They are related to the segment size, overlapping, statistical parameters selection (features), and the clustering method. According to some earlier work and our experience, it may be necessary to use the further processing of features. In the next section, an original methodology to segment historical data from condition monitoring system is presented in detail.

In the case of non-Gaussian noise, many theoretical assumptions cannot be fulfilled, so a number of techniques cannot be applied. This happens because the high content of outliers in the data negatively affects the calculation of many statistics, especially the ones related to variance.

4. Methodology

The general idea of the methodology is presented in Figure 2. It is assumed that the historical data set from condition monitoring systems contains three regimes that follow the model discussed in [49,50]. The degradation model consists of deterministic and random parts. Using a moving window, one may consider the original data in the window as independent and identically distributed (i.i.d.) observations, so simple statistical descriptors can be used to characterize their local properties. To increase resolution in the time domain, one may use overlapping. It may be helpful to precisely detect the regime change point.

In many papers, it is shown that simple statistics are easy to use and may provide important information; however, they are usually correlated, so dimensionality reduction is recommended. By the appropriate transformation of extracted features, one may reduce the number of novel features and still preserve original information. It is proposed to use principal component analysis (PCA) here. In general, the PCA is used to reduce highly dimensional data to a data set of a lower size [51]. However, in some applications, the PCA is used to visualize reduced features in 2D or 3D space, so projection is into PC1-PC2 or PC1-PC3 space. We will follow this idea here as well to visualize the distribution of data in PC space and to understand the results of clustering. To find these three regimes, we use extracted features and several most popular algorithms for clustering implemented in the Matlab environment.

In the case of really high correlation in input data space, it may happen that PC1 contains almost all information. Then, for 1D time series (PC1 corresponds to input data acquired in time), simple probability density function estimation may be enough to find threshold values that will be able to divide the input data into three regimes. The final step of the procedure is related to the validation of clustering. We present the results of clustering using different colors applied to time series. In Figure 3, more detailed explanation regarding moving window and statistical parameter extraction is presented.

4.1. Segmentation and Descriptive Statistics Used as Features

In the parametrization, we calculate the appropriate statistics calculated for the data from the moving window with size w and overlapping o. The window size cannot be too small because the parameters extracted for a small data set will be very sensitive to any outliers or even small variations. From the other hand, it cannot be large, as the resolution in the time domain will be poor and the precision of the regime change detection will be limited. The parameters w = 50 [samples] and o = 80 [percent] are selected experimentally. For each segment, a set of statistical parameters is calculated. There are many statistical features that can be extracted from time series based on the proposed methodology. In this paper, initially seven popular parameters are tested (maximum, sample median, sample mean, sample standard deviation (STD), sample kurtosis, sample skewness, and root mean square (RMS) of the data) and used as the main features. However, after some preliminary analysis, only three of them are selected. This issue is discussed in the next parts.

The mathematical formulations of the used statistics are presented in

Table 1. Time domain features.

Feature	Formula
Max value	M = max($x_i$)
Sample median	$\left\{\begin{array}{c}{x}_{(n+1)/2},n\%2=1,\hfill \\ \frac{x_{(n/2)}+x_{(n/2)+1}}{2},n\%2=0,\hfill \end{array}\right.$
Sample mean value	$\overline{x}=\frac{1}{N}{\displaystyle \sum _{i=1}^N}{x}_i$
Sample standard deviation (STD)	$\widehat{\sigma }=\sqrt{\frac{1}{N-1}{\displaystyle \sum _{n=1}^N}{(x_i-\overline{x})}^2}$
Sample kurtosis	$\mathrm{K}=\frac{1}{N}{\displaystyle \sum _{i=1}^N}\frac{{(x_i-\overline{x})}^4}{{\widehat{\sigma }}^4}$
Sample skewness	$\mathrm{Sk}=\frac{1}{N}{\displaystyle \sum _{i=1}^N}\frac{{(x_i-\overline{x})}^3}{{\widehat{\sigma }}^3}$
root mean square	$\mathrm{RMS}=\sqrt{\frac{1}{N}{\displaystyle \sum _{i=1}^N}{x}_i^2}$

Where the symbol % indicates the modulo division, and the result is a remainder.

, where $x_1,x_2,\dots ,x_N$ is the analyzed sample and N is the total number of observations.

4.2. Principal Component Analysis

Principal component analysis (PCA) in the mathematical definition [51] is an orthogonal linear transformation that takes data to a new coordinate system. Assuming that a given multidimensional data set is arranged as N samples with M variables, it can be expressed as a point cloud in M-dimensional space. The aim of PCA is to translate the location and orientation of the new coordinate system so that the variance is maximized along the new dimensions. It is performed in a way that the most significant variance of the data is on the first coordinate axis, and the second-largest variance is on the second coordinate axis, and so on. The idea is visualized in Figure 4.

Principal component analysis can reduce the data dimension, thus preserving the components of the data set that have the most significant effect on variance. For example, Figure 4 presents that the first principal component (the longer arrow) describes most of the variance in the data. Hence, if one extracts only the coordinates projected on this axis, one would obtain a majority of information described by the entire data set (or rather the percentage of the information corresponding to the relative content of the variance described by this new dimension).

By utilizing the singular value decomposition (SVD), the principal components can be calculated based on N observations of M-dimensional data stacked into a matrix $X\in {\mathbb{R}}^{N\times M}$

$$\frac{1}{\sqrt{N-1}}\mathbf{X}=\mathbf{U}\mathsf{\Sigma }{\mathbf{V}}^T,$$

(1)

where $U\in {\mathbb{R}}^{N\times N}$ and $V\in {\mathbb{R}}^{M\times M}$ are unitary matrices and $\mathsf{\Sigma }\in {\mathbb{R}}^{N\times M}$ contains the nonnegative real singular values of non-increasing magnitude (${\sigma }_{pc1}\ge {\sigma }_{pc2}\ge \cdots \ge {\sigma }_{pcM}\ge 0$). Principal components are the orthonormal column vectors of the matrix V, and the variance of the i-th component is equal to ${\sigma }_{pci}^2$.

4.3. Kernel Density Estimation

The probability distribution function (pdf) can be estimated using the kernel density estimator [53,54]. For argument x, the estimated density is given by

$${\widehat{f}}_h\left(x\right)=\frac{1}{Nh}\sum _{i=1}^{N}W\left(\frac{x-x_i}{h}\right),$$

(2)

where $x_1,x_2,\dots ,x_N$ are observations, $W(·)$ is the kernel smoothing function, N is the sample size, and h is the bandwidth. In this paper, a Gaussian kernel is used. The Gaussian kernel is the most common one used in various applications. It corresponds to the Gaussian probability density function; however, it is also used to estimate the probability density functions from other distributions. The Gaussian kernel has the properties of having no overshoot to a step function input while minimizing the rise and fall time. The value of the bandwidth is obtained using the so-called Silverman’s rule of thumb [54]. For the Gaussian kernel, the optimal choice for h (that is, the bandwidth that minimizes the mean integrated squared error) is

$$h={\left(\frac{4{\widehat{\sigma }}^5}{3N}\right)}^{\frac{1}{5}}\approx 1.06\widehat{\sigma }{N}^{-1/5},$$

(3)

where $\widehat{\sigma }$ is the sample standard deviation and N is the number of observations.

4.4. Cluster Analysis Techniques

In cluster analysis, many various algorithms are available that are characterized with different principles of operation and suitable for different scenarios. We test several popular methods using implementations prepared for Matlab. Finally, we have decided to use three algorithms providing the most promising results. Here, we recall three of the most suitable in our context.

• K-means clustering is a method of vector quantification that is originally derived from signal processing, and it is famous approach for clustering in data mining [55,56,57,58]. K-means clustering is used to decompose the n observations into k clusters whose observations belong to a cluster with its closest mean. According to the set of observations $x_1,x_2,\dots ,x_N$ where each observation is a M dimension vector. The target of K-means clustering is to divide N observation to $k<=N$ collection $S=(S_1,S_2,\dots ,S_k)$ such that the sum of the squares of the difference from the mean (i.e., variance) is minimized for each cluster.
• BIRCH (balanced iterative reducing and clustering using hierarchies) is one of the fastest clustering algorithms, introduced in refs. [59,60,61]. The main advantage of BIRCH is that it clusters incrementally and dynamically, attempting to produce the best quality given the time and memory constraints, with the requirement of only a single scan of the data set. However, it needs to specify the cluster count as an input variable. Additionally, BIRCH clustering is used in engineering applications. Lu et al. [62] introduced automatic fault detection based on BIRCH.
• Gaussian mixture modeling (GMM) is one of the popular methods used for unbalanced data clustering. The GMM is a probabilistic model that is based on the assumption that M-dimensional data X are arranged as a number of spatially-distributed Gaussian distribution modes with unknown parameters $\mu$ (list of M-dimensional means) and $\mathsf{\Sigma }$ (list of $M\times M$ covariance matrices).
  The expectation–maximization (EM) algorithm is used to estimate the parameters of GMM, thus allowing to cluster the data [63,64,65,66,67,68]. This method can be divided into two parts. At first, the expectation step (E-step) is used to estimate the probabilities for every point to be assigned to every cluster. Then the maximization step (M-step) is utilized to estimate the distributions based on the probabilities from E-step. Those two steps are iterated for a given number of iterations or until convergence.

5. Simulated Data Analysis

In this section, we apply the proposed procedure to simulated data. There are two cases: deterministic trend describing the degradation process mixed with (a) Gaussian and (b) non-Gaussian noise. The model is described as follows:

$$HI\left(t\right)=\left\{\begin{array}{c}a\hfill \\ b·t\hfill \\ c·\exp (d·t)\hfill \end{array}\right.\left.\begin{array}{c}t\le 1000\hfill \\ 1000<t\le 1600\hfill \\ t>1600\hfill \end{array}\right\}+\left\{\begin{array}{c}{\sigma }_1·N\left(t\right)\hfill \\ {\sigma }_2·t·N\left(t\right)\hfill \\ {\sigma }_3·\exp \left(t\right)·N\left(t\right)\hfill \end{array}\right.\left.\begin{array}{c}t\le 1000\hfill \\ 1000<t\le 1600\hfill \\ t>1600\hfill \end{array}\right\},$$

(4)

where $a,b,c,d$ are constant parameters that are related to deterministic parts, and ${\sigma }_1,{\sigma }_2,{\sigma }_3$ are parameters responsible for the scales of noise parts. Moreover, $\left\{N\right(t\left)\right\}$ is a noise.

Internal and external Gaussian noise is present in every real system due to various reasons. Unfortunately, the case becomes much more complicated when additional non-Gaussian behavior is present in the signal. Such a model of the signal is inspired by real long-term data we collected from various machines. The source of non-Gaussian noise may be related to machine design, the process related to machine operation, electromagnetic interference, etc.

5.1. Signal Simulation for Gaussian and Non-Gaussian Noise

The variance for Gaussian noise is time varying; its increase may be described as a linear or exponential function according to Equation (4). Simulation of degradation data in the presence of Gaussian noise is shown in Figure 5.

For a non-Gaussian case, the symmetric $\alpha$-stable distribution is selected as an example of heavy-tailed, non-Gaussian distribution [41,69,70,71,72].

The $\alpha$-stable distribution is defined by its characteristic function, and it is characterized by four parameters: $\alpha$ (stability), $\beta$ (skewness), $\sigma$ (scale) and $\mu$ (location). However, for the symmetric case, it is assumed $\beta =\mu =0$, and the corresponding characteristic function takes the form

$$\begin{array}{c}\hfill E\left[e^{itX}\right]=\exp \left\{-{\sigma }^{\alpha }{\left|t\right|}^{\alpha }\right\}.\end{array}$$

(5)

The $\alpha$ is known as the stability index and takes values in the interval $(0,2]$. It should be noted that the $\alpha$-stable distribution reduces to the Gaussian distribution when $\alpha =2$. In the case of decreasing $\alpha$ value, the distribution is significantly different from the Gaussian distribution [73]. In the presented simulation study, we assume the stability index is equal to 1.8.

Exemplary simulated data with non-Gaussian distributed noise are shown in Figure 6.

5.2. Extraction of Features for Simulated Signal for Gaussian Noise Case

In Figure 7, seven initially extracted features for Gaussian noise case are plotted.

Based on visual inspection of each statistical feature, three features are selected for further processing, namely STD, RMS, and mean value. They are selected because their shapes are the most similar to the original curve. In Figure 8, a 3D plot of these three features is presented. One may notice that three potential clusters are possible. In the Figure 8 3D plot, one can notice internal dependence/correlation in the data, thus a PCA is applied. Indeed, as one can see in Figure 9, PC1 contains nearly 100 % information. In Figure 10, a comparison of three original features and PC1–PC3 features are presented. One may see a strong correlation between the original features and PC1.

As mentioned above, as original features are highly correlated and the PC1 contains most of the information, the clustering can be done simply. In Figure 11, one can see the trajectory of PC1 (left) and its probability density function. One can easily notice three modes related to three regimes: the first one with amplitude less than 0 is related to regime (good condition), second one with amplitude more than 0 and less than 10 (warning condition) and samples with amplitude above 10 could be considered as regime 3. In Figure 12, the result of clustering in PC1–PC2 space is presented. The degradation development goes from the left side (green cluster) through points cloud in yellow color to several point constituting “Alarm” cluster. As one can see EM, BIRCH and K-means detected three classes with different results. For example, the EM algorithm detected the first and second regimes earlier than other approaches, while K-means and BIRCH discovered the last regime the same. However, the result for the second regime is different. Moreover, BIRCH and K-means recognized more (than EM) points as “Good Condition”, i.e., they may have some trouble recognizing the border between class 1 and class 2 (good and warning) properly.

Finally, in Figure 13, the results of segmentation based on selected clustering algorithm are presented. In Figure 13, one may find a confirmation that K-means and BIRCH recognized too many points as a “good condition” cluster. The change point between the good and warning segment is much later than for the EM algorithm. Hence, one may conclude that the EM algorithm is the best solution here. The exact values of changing point (CP) are presented in

Table 2. Final changing points in case of Gaussian noise.

Algorithm	1st Point	2nd Point
EM	998	1500
K-Means	1240	1570
BIRCH	1390	1580

.

5.3. Extraction of Features from Simulated Signal for Non-Gaussian Noise Case

By analogy, as for the Gaussian case, now the procedure is applied for simulated data with non-Gaussian noise. In Figure 14, all features are presented. It is clear that some of them are not suitable, so they are not considered for later stages. In Figure 15 (left top panel), three selected features are presented. One may conclude that the concept of moving windows and selected statistics as features works properly for Gaussian noise. However, for non-Gaussian noise, when a single outlier occurs, the effect generated by moving window and overlapping produces more artifacts than the original data contain. Hence, while statistics, such as sample mean, are sensitive to a single outlier, also the classical parameters responsible for variance (as sample variance) are not recommended for data with impulsive noise.

In Figure 16, features are plotted in 3D space. Similar to the Gaussian case, regime 3 is clearly noticeable; however, regimes 1 and 2 can be distinguished as well. The impact of distortion of the features does not appear to be significant. In the next step, the data are processed by PCA. Again, the first component (PC1) contains the majority of information; see Figure 17. In Figure 15, one can see that the shape of PC1 is similar to the original data, and PC2 may be useful to recognize a third regime.

In Figure 18, the PC1 time series and its kernel density is presented. One may notice a strong artifact around T = 100. However, based on the empirical probability density function, it is possible to establish threshold values to identify regimes in PC1 (thresholds at c.a PC1 = 0 and PC1 = 10).

In Figure 19, the final results of clustering in the PC domain are demonstrated. The degradation process starts from the healthy stage (green points) to the warning and alarm stages (yellow and red points). As it can be seen in Figure 19, K-means and BIRCH detected regimes as the same while EM algorithm discover the second regime around (−3, −2).

Furthermore, K-means and BIRCH can separate regimes representing the warning and alarm stages, i.e., the yellow and red points. The results of EM for separating yellow and red points are acceptable. Finally, in Figure 20, the last results for simulated data with non-Gaussian noise are presented. It is clearly seen that the results include some mixture of classes that could not be seen in simulations with Gaussian data. Additionally, Figure 20 confirms that the results of clustering with K-means and BIRCH algorithms are correct. However, both of them detected the second stage with significant delay. The delay for EM clustering is less than K-means and BIRCH. The exact values of changing point (CP) are presented in

Table 3. Final changing points in case of non-Gaussian noise.

Algorithm	1st Point	2nd Point
EM	1050	1610
K-Means	1240	1590
BIRCH	1252	1592

.

6. Real Data Analysis

In this section, we present results of application of the proposed procedure to real data sets. Again, two data sets are considered. One of them is related to FEMTO data—one of the most popular data sets widely used as benchmark. It could be interpreted as rather Gaussian case. The second real example is related to wind turbine bearings data. It contains strong non-Gaussian components.

6.1. Real Data with Almost Gaussian Noise

The FEMTO data set [74] was acquired by Franche-Comté Electronics Mechanics Thermal Science and Optics–Sciences and Technologies institute from a PRONOSTIA platform. The test rig is presented in Figure 21. The data set was shared during the prognosis challenge at the conference IEEE PHM 2012. The data set contains 17 historical long-term feature sets describing the degradation of bearing. Two accelerometers and a temperature sensor were used to acquire acceleration and temperature. The speed of the shaft was kept stable during the test. It is assumed that the failure of the bearing occurs when the amplitude of the vibration signal has arrived above 20 g. This data set was used in many publication for segmentation [75,76,77,78,79], construction of the health index [80,81,82,83,84,85,86,87] and predicting RUL [88,89,90,91,92,93].

As one can see, this data set perfectly follows the idea of three regimes: for time 0 to c.a. 12,000, it is nearly flat, then up to t = 27,000 it shows linear increase and then rapid growth Figure 22. Some noise, especially in the middle regime, is also seen; additionally, some small outliers can be noticed. Thus, we consider it as a trend with nearly Gaussian noise. In Figure 23 features extracted using the proposed methodology are plotted. As one can see, RMS and mean value are very similar, and STD reacts significantly only for the third regime. Unfortunately, kurtosis and skewness seem to be not related to degradation. Three features are used for further analysis: STD, RMS and mean value.

In Figure 24, the selected features are plotted together as a 3D plot. As one can expect, data corresponding to the third regime are pretty different compared to regimes 1 and 2, so it can be a bit challenging to recognize regimes 1 and 2.

Next, the PCA is applied. As presented in Figure 25, PC1 contains over 95% of information.

In Figure 26, the results of the PCA analysis are presented. Three original features are plotted together with three principal components. It is clear that the PC1 is following the original features, while PC2 and especially PC3 do not contain too much information. As PC1 is the most informative, one can use the probability density function (PDF) to set up thresholds for regime change point detection. As mentioned above, as original features are highly correlated and the PC1 contains most of the information, the clustering can be done. In Figure 27, one can see the trajectory of PC1 (left) and its PDF. One can easily notice three modes related to three stages: the first one with amplitude less than 0 is related to the healthy (good condition) stage, the second one with amplitude more than 0 and less 0.2 (degradation stage, i.e., warning), and samples with amplitude higher than 0.2 could be considered as critical stages (Alarm).

In Figure 28, the results of the selected clustering techniques are presented. The data with high dispersion correspond to regime no. 3 (red color). Cluster in green color corresponds to the healthy stage. The last one is medium degradation regime (yellow). In general, the EM algorithm provides the most effective result; however, it is not perfect. BIRCH and K-means provide similar results.

In Figure 29, the final result of segmentation is presented. Original data divided into three regimes are plotted with different colors associated to a given regime. As shown in Figure 29, the BIRCH and K-means discovered the second change point much later than what really exists. Additionally, the first change point, which should start around T = 12,000, is detected significantly later by the K-means and BIRCH algorithms. The EM algorithm can detect the third regime (critical stage) correctly, but the first and second regimes are mixed together. It should be highlighted that the difference between the end of regime 1 and the beginning of regime 2 is very small, so there is no surprise that precision is limited here.

However, it is a bit surprising that BIRCH and K-means have some problems with the proper recognition of regime 3. To summarize, the best result is obtained by EM. Exact values of changing point (CP) are presented in

Table 4. Final changing points for the FEMTO data set.

Algorithm	1st Point	2nd Point
EM	13,170	27,120
K-Means	19,780	27,400
BIRCH	20,200	27,800

.

6.2. Real Data with Strong Non-Gaussian Noise

In this section, we provide an example of real data with strongly non-Gaussian noise. This data set contains diagnostic features based on collected vibrations from wind turbine [94] bearings; see Figure 30.

It should be mentioned that this data set has been used for prognosis in several works [94,96,97,98]. In this case, for the construction of the HI, the method proposed in ref. [94] is used. The health index extraction procedure is illustrated on Figure 31. For details of the method, we refer the readers to [94]. The values of inner race energy for a group of measurements for one particular speed (high speed) are demonstrated in Figure 32. In Figure 33, all calculated features are presented. Again, some of them seem not to be useful (does not follow the shape of original data), so they will be ignored for the next step of analysis. In Figure 34, selected features are presented in a 3D plot. Definitely, they are strongly correlated, so the utilization of the PCA is suggested. In Figure 35, the importance of principal components is presented. More than 90 % of the informativeness is included in PC1. In Figure 36, original features and new features (i.e., Principal Components) are presented. One can see that the shape of the PC1 is similar to the original features, while the behavior of PC2 and PC3 is different. Based on PC1 data, the PDF is estimated. It is presented in Figure 37. Threshold values are established experimentally based on PDF of PC1. The right subplot presents segmentation results using colors.

Based on this fact, the wind turbine data set includes many spikes; it is not easy to select the threshold. However, it can be selected approximately: the first one with an amplitude less than 0 is related to the healthy stage, the second one with an amplitude more than 0 and less than 0.2 (degradation stage), and samples with an amplitude higher than 0.2 could be considered as critical stages. Next, three selected features are subjected to clustering algorithms; see Figure 38. All of the algorithms identified three clusters; however, borders for BIRCH and K-means seem to be more clear than for EM. The final results, i.e., clustering-based segmentation for real non-Gaussian data, are presented in Figure 39. As it was concluded for Figure 38, the EM algorithm detects regime 2 too early. More or less, the second regime is discovered by BIRCH and K-means equally; however, the result for detecting the last regime is different. In Figure 39, the final result for the wind turbine data set is presented. Original data divided into three regimes are plotted with different colors associated to the given regime. As shown in Figure 39, the EM could not detect the third regime (critical stage) properly. The other clustering (BIRCH and K-means) approaches are able to divide data into three regimes in a similar way. For this case, the exact changing points are undefined; see

Table 5. Final changing points for wind turbine data set.

Algorithm	1st Point	2nd Point
EM	Undefine	Undefine
K-Means	Undefine	Undefine
BIRCH	Undefine	Undefine

.

7. Conclusions

In the paper, a simple procedure for long-term diagnostic data segmentation is proposed. The procedure is developed to find the regime change point, i.e., borders between three regimes corresponding to three phases of machine life: good condition, slow linear degradation and fast (exponential) degradation.

To achieve the goal, we propose the statistical parameterization of the data performed locally, i.e., using moving short window and then simple techniques for clustering.

It is found that for Gaussian distribution of noise in the data, the efficiency of the method is good—we are able to detect regime change point. As the difference between good condition and beginning of slow degradation is not really significant, border detection is a bit biased. It is not critical (the machine is still in almost good condition).

Unfortunately, non-Gaussian distribution of the noise influences the procedure in several ways. First, the idea of moving window is not really suitable because one outlier will provide several features with bias. Moreover, simple statistics suitable to describe data with Gaussian noise are not optimal for non-Gaussian data. For example, the mean value will be affected by outliers; if the window is short, this influence may be significant. Standard deviation is even more sensitive to outliers, and some heavy-tailed processes cannot use variance at all as it is not defined for such processes. Finally, PCA should not be used in the classical form for heavy-tailed data because it maximize variance (model will follow outliers) and all these steps will impact regime change point detection. This implies a need to search for new approaches. For future work, it is planned that used statistics may be replaced with a robust version (median instead of mean, robust scale instead of variance, etc.). There are many solutions for dimensionality reduction for non-Gaussian data sets. Additionally, clustering methods for non-Gaussian data are available.

The main purpose of this work is to highlight (by simulated and real data analysis) that the non-Gaussian character of the random component in our long-term data should be investigated and evaluated also during the segmentation step before data modeling for machine health prognosis.