Fault Diagnosis in Induction Motors through Infrared Thermal Images Using Convolutional Neural Network Feature Extraction

Abstract

The development of diagnostic systems for rotating machines such as induction motors (IMs) is a task of utmost importance for the industrial sector. Reliable diagnostic systems allow for the accurate detection of different faults. Different methods based on the acquisition of thermal images (TIs) have emerged as diagnosis systems for the detection of IM faults to prevent the further generation of faults. However, these methods are based on artisanal feature selection, so obtaining high accuracy rates is usually challenging. For this reason, in this work, a new system for fault detection in IMs based on convolutional neural networks (CNNs) and thermal images (TIs) is presented. The system is based on the training of a CNN using TIs to select and extract the most salient features of each fault present in the IM. Subsequently, a classifier based on a decision tree (DT) algorithm is trained using the features learned by the CNN to infer the motor conditions. The results of this methodology show an improvement in the accuracy, precision, recall, and F1-score metrics for 11 different conditions.

1. Introduction

In the last decade, rotatory machines such as induction motors (IMs) have played an important role in industrial development. Industries such as manufacturing, transportation, and fabrics benefit from the high-efficiency generation of IMs [1,2,3]. However, IMs are subject to harsh working conditions such as long working periods, mechanical and electrical stress, overloads, abrasion, and unbalanced loads, resulting in premature deterioration and motor failure. For this reason, monitoring systems have been developed to prevent IMs from being damaged or to avoid waste of resources [4,5]. Early detection of faults or abnormal states in IMs, including stator faults, rotor electrical faults, and short-circuit faults, can improve the lifespan of the motor, generating maximum productivity and minimum downtime.

In recent years, a large number of methodologies have been proposed for fault detection in IMs. These methodologies can be divided into two different sets, namely invasive methodologies and non-invasive methodologies [6,7]. On the one hand, within invasive methodologies, signal acquisition is the most used technique for detecting failures in IMs. Torque and current are the most commonly measured signals [8,9,10]. However, due to their complexity, these methodologies are not suitable for harsh work environments, since it could be difficult to identify faults in IMs at early stages. Moreover, these methodologies are usually expensive and can cause fatal injuries during installation [11,12]. On the other hand, non-invasive methodologies such as thermal analysis (TA) emerge as an alternative for detecting faults in IMs [13]. The main objective of TA is to capture patterns through thermal images (TIs) and infer the motor conditions. IM failures often cause an increase in temperature, so it is possible to diagnose motor conditions based on thermal patterns. The framework followed by the methods that use TIs involves improving the output image via pre-processing to detect the region of interest (ROI) [14,15,16]. Subsequently, different segmentation techniques, both manual and automatic, are used to infer the motor conditions [17]. In the first case, within the manual segmentation, in [18], TIs were manually segmented, and the hottest points were detected to determine the failures in the IM. In [14], first- and second-order statistical features were extracted to select the most outstanding ones using a linear discriminant algorithm (LDA) [19]. Subsequently, a multi-layer perceptron (MLP) wass used to infer the motor conditions. In [20], a histogram-based technique was used to extract features from the TIs. Then, an MLP was used to infer the faults present in the motor. Alternatively, within the automatic segmentation methods, in [21], the watershed technique was implemented to find the area of interest within the TIs. Subsequently, a neuro-fuzzy classifier was implemented to categorize faults in the IM. In [17], thermal images were segmented into three different zones using the scale-invariant feature transform (SIFT) method. Then, temperature feature was extracted in each zone. Finally, this feature was used to build a classifier and infer the motor conditions. In [22], thermal images were first categorized into two classes, namely cold and hot, using a decision tree. Subsequently, the region of interest was extracted using the block-wise method and the random forest algorithm. Finally the random forest was trained to infer 11 different faults present in the IM.

Although the aforementioned methods achieve a reasonable classification rate in automatically classifying IM faults, there are still unresolved practical problems. First, although the extraction of statistical features provides useful information for the automatic identification of faults in IMs, the acquisition of TIs under different environmental conditions can generate different statistical features. Therefore, different failures presented in the IM could have statistical features in common, generating misclassification problems. Secondly, although the extraction of the ROI allows for segmentation in the study area, this area is compromised when the raw image presents noise, which can generate a false ROI, triggering misclassification. Thirdly, recent classification methods used in IMs are based on unbiased datasets. This generates false ranking indices so that such methods cannot not perform well in practical environments. To address the aforementioned problems, a new system based on CNN and DT classifiers is presented. Using the dataset presented in [22,23], two subsets (training and testing sets) are created to train, tune, and evaluate the final model. First, the training set is used to tune a CNN. The tuning process consists of fitting the training set to the CNN model. Two outputs are part of the CNN model. One output, composed of 11 nodes, is used to adjust the weights in the network. The second output is used as a feature vector to fit the DT. Once the CNN is trained, the training set is evaluated again on the CNN to extract the most salient features of each of the faults. Then, these features are used to train and tune a DT classifier. Finally, the DT classifier is evaluated using the testing set, showing improvements in metrics such as accuracy, precision, recall, and F1 score. Although this technique has been applied in signal and image processing areas [24,25,26], it has not been implemented in detecting faults in induction motors. The main contribution of the proposed model is that there is no need to extract features from TIs manually. The features are automatically extracted by the CNN. Moreover, the proposed model is resistant to noise presented in the image, which makes the model more suitable for harsh environments. Thus, the model efficiently classifies 11 different IM fault conditions.

The remainder of this paper is structured as follows. Section 2 describes the methods used to infer the motor conditions based on TIs. The experimentation and the obtained results are discussed in Section 3. Finally, Section 4 describes the conclusions of the work.

2. Proposed Methodology

This section describes the proposed methodology for classifying IM faults in 11 interest classes. An overall description of the development of this methodology is presented in Figure 1. In this figure, it can be observed that the proposed method comprises six stages, namely image input, data augmentation, creation of training and testing sets, extraction of features with a CNN, training the DT based on extracted CNN features, and performance evaluation. In the first and second stages, the dataset proposed in [22] is subjected to random flip and random rotation in order to homogenize the classes in the dataset. In the third stage, the training and testing sets are created to train, tune, and evaluate the performance of the final model. Once the datasets are created, the training set is used in stage four to train a CNN that allows for the extraction of the most relevant features from each fault. Subsequently, in stage five, these features are used to train a DT classifier and infer the condition of the IM. Finally, the performance of the DT classifier is evaluated using the testing set. More details are presented in the following subsections.

2.1. Dataset Description

In this study, the dataset proposed in [22,23] was used to develop the suggested methodology. This dataset contains 369 images distributed across 11 different conditions. Each image has dimensions of $360\times 240$ in RGB (red, green, and blue) format. Figure 2 shows the different conditions present in the dataset. From this figure, it can be observed that the dataset is composed of a healthy condition, 8 different inter-turn faults (ITFs), the combination of windings and stuck rotor faults, and cooling fan faults.

Table 1. Image distribution presented in the dataset. According to the authors of [22,23], the % symbol represents the rate of short circuit in each phase. The numbered phases represent the phases in which the short circuit occurred.

Class	Images	Dimensions	Format
Cooling	28	360 × 240 × 3	BMP
Rotor	30	360 × 240 × 3	BMP
50% stator 2-phase	38	360 × 240 × 3	BMP
50% stator 1-phase	35	360 × 240 × 3	BMP
30% stator 3-phase	42	360 × 240 × 3	BMP
30% stator 2-phase	38	360 × 240 × 3	BMP
30% stator 1-phase	37	360 × 240 × 3	BMP
10% stator 3-phase	31	360 × 240 × 3	BMP
10% stator 2-phase	31	360 × 240 × 3	BMP
10% stator 1-phase	34	360 × 240 × 3	BMP
Healthy	25	360 × 240 × 3	BMP
Total	369	-	-

describes the set of images presented in each class.

2.2. Data Augmentation

In machine learning, data augmentation is a technique used to create artificial data from existing data [27]. The main goal of data augmentation is to homogenize the dataset. In this study, special attention is paid to unbalanced classes. This is because unbalanced datasets can generate misclassification. According to Table 1, the number of TIs presented in each class is lower than 30% stator 3-phase fault (42 thermal images). It is concluded that the dataset is not balanced. To address this issue, random flips (horizontal and vertical) and random rotations are used to balance the dataset according to the class with more TIs (this is the 30% stator 3-phase class). Figure 3 shows an example of the transformations used in the 50% stator 2-phase fault.

Once the dataset is balanced (462 thermal images, 42 images per class), two stratified sets are created, namely the training set and the testing set. The training set is formed by 369 (80% of the balanced dataset) TIs, while the testing set is composed of 93 (20% of the balanced dataset) TIs.

Table 2. Data distribution used to train and evaluate the performance of the final model.

Dataset/Class	Training Set	Testing Set	Total
Rotor	33	9	42
Healthy	34	8	42
10%-stator 3-phase	34	8	42
50%-stator 1-phase	33	9	42
50%-stator 2-phase	34	8	42
30%-stator 2-phase	33	9	42
10%-stator 2-phase	34	8	42
10%-stator 1-phase	33	9	42
30%-stator 3-phase	34	8	42
30%-stator 1-phase	34	8	42
Cooling	33	9	42
Total	369	93	462

shows the data distribution through each set. Finally, all images are resized to $250\times 250\times 3$ to fit in the CNN model.

2.3. Feature Extraction Using Convolutional Neural Network

In the classification process, feature extraction is a crucial step for the development of a predictive model. Relevant features need to be extracted to achieve high classification metrics. Traditional feature extraction methods are based on statistical features, while a CNN can learn features from raw images, achieving a better performance [28,29]. In this study, a CNN was constructed to learn features from the raw TIs. Thus, the proposed CNN is formed by two blocks, namely the feature-learning block and the classification block. Figure 4 shows the structure of the proposed CNN model.

In the feature-learning block, different convolutional operations are used to extract the most reliant features from each IM fault. The input data of this block are the raw TIs, which are a 3-dimensional matrix with a size of $250\times 250\times 3$. According to Figure 4, the featuring-learning block is formed by 5 blocks of 2-dimensional convolutional layers (2D convolutions) [30], 5 blocks of max-pooling layers [31], and rectified linear units (ReLUs) of activation functions [32].

Once the feature-learning stage is executed, the extracted feature vectors are input into the classification block. The classification block is formed by the flattening layer, one dense layer with 512 units and a ReLU as an activation function, a dropout layer [33] with a 0.5 frequency rate, one dense layer with 11 units, and a softmax activation function [34]. Finally, the loss function, the optimizer, the learning rate, and the batch size are set to sparse categorical cross entropy, the Adam optimizer, 0.001, and 16, respectively. Appendix A.1 describes the CNN training process.

Once the CNN is trained and evaluated, the feature-learning block is used to extract features from the training set. The main goal is to replace the classification block with a different classifier and maximize the performance in the evaluation process. In this study, the classification block was replaced with a decision tree (DT) classifier [35]. This is because fitting small datasets in a CNN can lead to overfitting [36]. For this reason, the classification block was changed to a model that can manipulate the information generated from small datasets—in this case, a DT [37]. The DT classifier was trained with the features generated by the CNN in the training set and evaluated using the features generated by the CNN in the testing set.

2.4. Decision Tree Classifier

In machine learning, a decision tree (DT) is a machine learning algorithm used for both classification and regression tasks [38,39]. The method consists of a root node and decision nodes. Based on the available features, both types of nodes perform evaluations to generate homogeneous subsets and create a final model [40]. To select the best attribute at each node, the Gini impurity gain and entropy gain methods act as splitting criteria for decision tree models. In this study, the Gini impurity gain was computed to select the best attribute at each node. Thus, the Gini impurity gain is defined according to Equation (1):

$$Gini=1-\sum _i^k{p}_i^2$$

(1)

where k denotes the samples from each class and $p_i$ represents the probability of samples belonging to class i. In this work, the DT classifier was trained with 80% of the data, corresponding to a vector composed of 369 samples and 512 features (the features extracted by the CNN) and evaluated by a testing set formed by 93 samples and 512 features.

3. Results and Discussion

3.1. Evaluation Metrics

In this study, all experimentation was performed on a computer with an AMD Ryzen 5600G to 3.9 GHz CPU, an NVIDIA GeForce GTX 1650 GPU with 4 GB, and 16 GB of RAM. The proposed models were run efficiently using PyTorch and scikit-learn frameworks [41]. During the implementation, different metrics were used to evaluate the performance of the model. Accuracy, precision, recall, and F1 score describe the evaluation process used in this study. Equations (2)–(5) illustrate the aforementioned metrics.

$$accuracy={\displaystyle \frac{TP+TN}{TP+FN+FP+TN}}$$

(2)

$$precision={\displaystyle \frac{TP}{TP+FP}}$$

(3)

$$recall={\displaystyle \frac{TP}{TP+FN}}$$

(4)

$$F1-score={\displaystyle \frac{2\left(precision\right)\left(recall\right)}{precision+recall}}$$

(5)

where $TP$ represents true positive, $TN$ represents true negative, $FP$ represents false positive, and $FN$ represents false negative.

3.2. Hyperparameter Tuning

To obtain an optimal CNN, different combinations of hyperparameters and structures were exhaustively analyzed. Because there is a large number of combinations of hyperparameters and structures, some of them were chosen empirically. The hyperparameters selected empirically were the learning rate, the dropout rate, the convolutional kernels, and the number of dense layers.

Table 3. Suggested models to infer faults in induction motors.

Model	Hyperparameters	Values	Activation Function
Model 1	Convolutional layer kernel size	16, 32, 64, 64, 128	ReLU
	Dense layer size	128, 11	ReLU,  Softmax
	Dropout rate	0.2	-
	Learning rate	0.01	-
Model 2	Convolutional layer kernel size	16, 16, 32, 64, 128	ReLU
	Dense layer size	256, 11	ReLU,  Softmax
	Dropout rate	0.3	-
	Learning rate	0.01	-

shows two proposed models to infer IM faults.

Once the experimentation with different hyperparameters was accomplished, the final CNN architecture was selected according to the validation error. The hyperparameters that obtain the lowest validation error are shown in

Table 4. Optimal CNN hyperparameters to infer induction motor faults.

Hyperparameter	Optimal Values
Learning rate	0.001
Dropout rate	0.5
Convolutional layer kernel size	32, 32, 64, 128, 256
Dense layer size	512, 11

. All the suggested models were trained for 20 epochs in a range of time of 1.2 to 1.9 min.

Figure 5 shows the validation loss obtained by the proposed models. According to Figure 5, the model with the least validation loss is model 3, with the hyperparameters shown in Table 4.

After the CNN architecture was selected, different decision tree classifiers were trained using the features learned by the CNN. Unlike convolutional neural networks, decision trees do not contain many hyperparameters to tune. This allows the model to be trained and tuned quickly.

Table 5. Hyperparameters used to train the DT.

Model Parameter	Value
Max deepth	7
Decision	Gini
Min samples to split	2
Splitter	Best

shows the hyperparameters that allowed the DT classifier to obtain the highest values for each metric.

For a clear vision of the improvement in classification rate achieved by the decision tree classifier using the CNN features, Figure 6 shows the confusion matrix generated by the CNN and the DT classifier. As shown in the figure, the classification index achieved by the decision tree classifier is higher than that achieved by the CNN.

3.3. Feature Extraction

To better understand the impact of feature selection, the Gradient-weighted Class Activation Mapping (Grad-CAM) algorithm was implemented [42], as discussed in the following analysis. The Grad-CAM method is a technique used to visualize the decision-making process generated by a CNN. The main objective is to calculate the gradient of the output class score concerning the feature maps of the last convolutional layer in the CNN; then, Grad-CAM determines the output class score. Subsequently, the feature maps are weighted using the calculated gradients, creating a heat map highlighting the input image’s main areas. Figure 7 shows the features extracted by the CNN that allow for the classification of the rotor class as a positive class.

As shown in the figure, the features obtained using the CNN are those that comprise the motor region. This removes unnecessary information from the image, leaving only the most salient features that allow the fault to be inferred.

3.4. Results

Using the classification results of the test set, the metrics established in Equations (2)–(5) are calculated. The results show that the proposed DT model with the CNN features reaches an accuracy of 98.0%, a precision of 98.0%, a recall of 98.0%, and an F1 score of 98.0% in a total of 93 samples distributed across 11 classes. These results show an improvement with the models represented in the state of the art. The details of the classification process are shown in

Table 6. Results obtained in the evaluation process.

Class	Accuracy %	Precision %	Recall %	F1 Score %
Rotor	100	100	100	100
Healthy	100	89	100	94
10%-stator 3-phase	100	100	100	100
50%-stator 1-phase	100	100	100	95
50%-stator 2-phase	88.84	90	89	94
30%-stator 2-phase	100	100	100	100
10%-stator 2-phase	100	100	100	100
10%-stator 1-phase	100	100	100	100
30%-stator 3-phase	100	100	100	100
30%-stator 1-phase	87.53	100	88	93
Cooling	100	100	100	100
Average	98.0	98.0	98.0	98.0

. Results shown in Table 6 indicate that the proposed model achieves good performance on a small dataset.

3.5. Model Comparison

In order to evaluate the performance of the proposed model, the effectiveness achieved by the proposed methodology was compared with the performance achieved by different models present in the literature. The results obtained by each model can be found in

Table 7. Comparison of the results obtained by applying the proposed methodology (NN, neural network; SVM, support vector machine; RF, random forest; RVM, relevance vector machine).

Works	Methods	Classes	Accuracy %	Precision %	Recall %	F1 Score %
Huda et al.  [14]	Statistical features and NN	2	82.4	81.1	84.6	82.8
Tran et al.  [49]	Image descompostion and RVM	4	100	-	-	-
Glowacz et al.  [46]	Image segmentation and NN	3	100	-	-	-
Bai et al.  [50]	Image enhancement and NN	6	92.5	-	-	-
Janssens et al.  [43]	Statistical features and SVM	8	88.2	90.6	88.2	89.5
Lozanov et al.  [44]	Statistical features and SVM	3	83.3	-	-	-
Karvelis et al.  [16]	Image segmentation and ML	5	91.4	89.7	90.2	90.4
Charitha et al.  [45]	Statistical features and RF	6	97.2	-	-	-
Najafi et al.  [22]	Image segmentation and RF	11	93.8	92.1	93.2	92.6
Proposed	CNN feature extraction and DT	11	98.0	98.0	98.0	98.0

. According to this table, the proposed model shows an advantage compared to the other models not only in the accuracy index but also in precision, recall, and F1 score. According to the comparison with the model proposed in [22], improvements of 5.2% in the accuracy metric, 5.9% in the precision metric, 4.8% in the recall metric, and 5.4% in the F1-score metric were achieved by the proposed model. In comparison with the methods proposed by different authors, the statistical feature extraction [14,43,44,45] techniques achieve good performance in IM fault classification. However, these methods are susceptible to noise presented in the image, preventing the achievement of good performance [28]. For this reason, the number of classes that the final model can classify decreases drastically. The proposed approach allows for the classification of 11 different classes, in comparison with [16,38,46], which only classifies a maximum of 6 different classes. This is because image processing is based on obtaining characteristic elements of an image based on the intensity of the pixels. However, a small variation in pixel intensity can cause completely different results [47]. Thus, the features extracted by the proposed CNN are a better option than the statistical feature methods. Therefore, the success of this methodology is due to the features extracted by the proposed CNN. Although the performance achieved by the proposed model is satisfactory, the metrics mentioned above can be improved if the size of the dataset increases. Increasing the size of the dataset makes the training data more diverse, preventing the model from misclassifying. The presence of more data results in better machine learning models [48].

4. Summary and Conclusions

In this work, a new framework based on a CNN and DT is proposed for the automatic detection of IM faults. The proposed CNN is used as a feature extractor to obtain the most salient features of different conditions of the IM. Once the features are extracted using the CNN, a DT is built to infer the motor conditions. In summary, the contribution of this research constitutes two parts. (1) The model uses raw thermographic images without filtering; this shows the model’s ability to handle images with noise. (2) The combination of a CNN as a feature extractor and a DT as a classifier improves the classification indices necessary to build a good classifier. In this research, the dataset proposed in [22] was used to measure the proposed model’s performance. The performance achieved by this model is 98% in the accuracy metric, 98% in the precision and recall metrics, and 98% in the F1-score metric. The proposed method achieves good classification metrics, low classification error, and automatic feature extraction, making it a potential candidate for automatic IM fault detection in rough environments. Future work should focus on achieving a higher performance index in the metrics mentioned earlier using assembled models.