Machine Fault Diagnosis: Experiments with Different Attention Mechanisms Using a Lightweight SqueezeNet Architecture

Abstract

As artificial intelligence technology progresses, deep learning models are increasingly utilized for machine fault classification. However, a significant drawback of current state-of-the-art models is their high computational complexity, rendering them unsuitable for deployment in portable devices. This paper presents a compact fault diagnosis model that integrates a self-attention SqueezeNet architecture with a hybrid texture representation technique utilizing empirical mode decomposition (EMD) and a gammatone spectrogram (GS) filter. In the model, the dominant signal is first isolated from the audio fault signals by discarding lower intrinsic mode functions (IMFs) from EMD, and subsequently, the dominant signals are transformed into 2D texture maps using the GS filter. These generated texture maps feed as input into the modified self-attention SqueezeNet classifier, featuring reduced model width and depth, for training and validation. Different attention modules were tested in the paper, including the self-attention, channel attention, spatial attention, and convolutional block attention module (CBAM). The models were tested on the MIMII and ToyADMOS datasets. The experimental results demonstrated that the self-attention mechanism with SqueezeNet achieved an accuracy of 97% on the previously unseen MIMII and ToyADMOS datasets. Furthermore, the proposed model outperformed the SqueezeNet attention model with other attention mechanisms and state-of-the-art deep architectures, exhibiting a higher precision, recall, and F1-score. Lastly, t-SNE is applied to visualize the features of the self-attention SqueezeNet for different fault classes of both MIMII and ToyADMOS.

1. Introduction

The modern economy mostly relies on industrial production. The production efficiency mostly depends on the status of the machines. An untimely equipment breakdown leads businesses to a situation in which there is no movement or activity at all. In addition, the repair time of faulty machines causes a negative impact on industrial production. With the advancement of artificial intelligence (AI), researchers have applied AI in machine fault diagnosis to deal with detection and prediction without human intervention [1]. These AI models are data-hungry, and different sensors [2,3], including acoustic, vibration, current, voltage, audio, and thermal sensors, are used to collect the machine’s environmental data.

The sensor’s data are in the time domain and one-dimensional. In traditional supervised machine learning models [4,5], extracted time and frequency domain features are utilized to train and test the support vector machine (SVM), logistic regression (LR), K-nearest neighbors (KNN), random forest (RF), and MLP neural networks (NNs). The advantage of machine learning (ML) models is that the model performance mostly depends on the characteristics of the training dataset, such as quality, quantity, and representation. Feature engineering deals with feature extraction and optimal feature selection. The main limitation of ML models is that they perform well on training datasets. Still, they show poor results for unknown datasets. As the industrial environment and machine faults are not always known, ML models are not suitable due to their feature dependency.

Like other applications, deep learning (DL)-based fault diagnosis is becoming a key issue due to its automatic feature extraction mechanism. The DL architecture consists of several sequential layers, where the output of the first layer will be the output of the second layer, and so on. The final layer is the dense or the output layer. The prior flattened layer converts the output of the convolutional layer into a single 1D vector to feed it as an input into the dense layer and then makes the decision based on the input 1D vector using an activation function. The models use 1D time domain sensor signals for training, validation, and testing [6,7] and automatically extract the deep features from the sensor data. The attention mechanism helps the model to focus on the most relevant parts or effective information of the input for prediction and classification [8]. Several attention mechanisms are applied in deep learning architectures, such as the Convolutional Block Attention Module (CBAM) [9], the Global Attention Block (GAB) [10], the Split-Attention module [11], the spatial/channel-wise attention block [12], and the Multipath Residual Attention Block (MRAB) [13].

In [14], Huang et al. show that a 2D-Convolutional Neural Network (CNN) can more accurately extract detailed information than a 1D CNN for time-varying signals by exhibiting a higher accuracy. As a consequence, several researchers have converted 1D sensor data into 2D images for utilizing 2D deep architectures. Islam et al. [15] generated a wavelet packet transform-based DDRgram for 1D acoustic emission (AE) signals. Instead of using 1D raw AE signals, they utilized DDRgram (1024 pixels) as an input in an adaptive deep CNN (ADCNN). The experimental results show that the DDRgram-based deep learning model with a small length outperforms the raw 1D signals by requiring less processing time in CPU/GPU architectures.

Shao et al. [16] used a pre-trained VGG16 network with ‘ImageNet’ (https://www.image-net.org/, accessed on 3 July 2024) weights to extract the lower-level features and fine-tuned the model with the generated fault images by applying wavelet transform. In [17], Wen et al. utilize a pre-trained VGG19 architecture with ‘ImageNet’ and fine-tune it with generated 224 × 224 RGB images by segmenting the 1D CWRU bearing dataset. A summary of the results of the pre-trained state-of-the-art deep learning architectures including ResNet50, InceptionV3, MobileNetv2, Xception, DenseNet121/169/201, and NASNets are presented in [18]. Besides cross-domain issues, the model complexity of transfer learning (TL) architectures is a big concern due to their higher trainable parameters. The computational complexity increases with the increase in the number of trainable parameters, and vice versa. To handle the computational complexity issue of the DL models, different researchers are applying lightweight deep architectures for fault diagnosis.

The domain adaptation issue is solved by implementing the TL on a 1D CNN [19,20]. Gou et al. used an Intelligent Maintenance System (IMS), CWRU, and a Railway Locomotive (RL) bearing dataset for implementing TL with pre-trained deep networks [19]. Among the three datasets, one or two datasets are used for pre-training and fine-tuned with another dataset. The deep TL model shows an average accuracy of 86% for 3000 epochs. Negative transfer is a concern in TL as, in some cases, the training of source tasks or pre-training leads to a degraded performance in fine-tuning. The probability of negative TL is increased when the pre-trained dataset is different from the fine-tuned dataset [21]. Han et al. [22] applied TL with various environmental datasets of the same machine to reduce discrimination discrepancies and negative TL. Li et al. [23] implemented self-supervised learning on a CNN using 2D grayscale images segmenting the 1D vibration sensor data. In [24], Berenji et al. use the SEU-bearing dataset to implement self-supervised learning. Wei et al. utilized a data transformation combinations (DCT)-SimCLR framework for implementing self-supervised learning [25]. In all models, pre-training is performed using a large unlabeled dataset and then fine-tuned with a small amount of label fault dataset.

1.1. Literature Review

In recent years, researchers have focused on texture-based feature extraction pre-processing methods along with DL architectures. In [26], Tang et al. and Islam et al. [27] generate grayscale images from vibration sensor data by subdividing large-size time series data, with pixel values ranging from 0 to 255 and a size of 64 × 64 and 89 × 89, respectively. In [27], a unique texture pattern is presented for each type of fault signal. In [28], the Hilbert transform operator is applied to generate 2D images for vibration sensor signals. As the Hilbert operator is a complex function, the authors used absolute values during image generation and then utilized those generated images as input in the deep CNN for classification. The limitation of the model is that the model is tested with the vibration signals dataset, which is not challenging due to its controlled environments (only varies noises). The Hilbert operator is computationally intensive for large data in real-time applications. In [29], Fan et al. propose a fault diagnosis method using generated gray texture images from vibration signals and a CNN with batch normalization. Although the model extracts features in variable load conditions including no-load, on-load, and unknown load, the dataset used in the experiment does not cover the industrial machine states.

In [30], Tran et al. utilize scalogram feature images by applying the continuous wavelet transform (CWT) to validate a convolutional attention neural network (CANN). In [31], Chen et al. propose a gate-CANN using long short-term memory and an attention module with a CNN. In both cases, CWT is applied to generate 2D time-frequency images from vibration signals. The fine-tuning of wavelet functions in the CWT is an issue as the overall performance mostly depends on the choice of wavelet function. The CWT is not suitable for industrial applications as it is noise sensitive.

In [32], Short-Time Fourier Transform (STFT) is applied to generate a spectrogram of vibration signals and then the generated spectrogram is utilized to validate a DL model using a stacked sparse autoencoder. In [33], Benkedjouh use STFT-based spectrograms for classifying faults using a traditional CNN. Du et al. [34] utilized STFT-based spectrogram images as an input in a transfer deep residual network (TDRN), where the TDRN makes the bridge between different working conditions using knowledge learned from one working condition to another.

Table 1. Summary of state-of-the-art models and their limitations.

Ref., Year	2D Image Generation Techniques and Dataset	Architectures	Limitations
[33], 2018	STFT-based spectrogram, vibration dataset	CNN	Although the STFT is an effective spectrogram for representing frequency components in time series data due to its simultaneous temporal and spectral information-capturing capabilities, it does not provide an optimal spectrogram resolution in both time and frequency domains.
[15], 2019	DDRgram, AE dataset	ADCNN	The model shows that the DDRgram-based input (1024 pixels) 2D adaptive deep CNN exhibited a higher diagnostic performance by requiring less processing time than the 1D raw signal for the AE dataset. Selecting the wavelet filter size when generating the DDRgram is a vital issue.
[35], 2021	RGBVI, vibration dataset	CNN	RGB vibration images are generated by segmenting the vibration dataset. Although the model shows improved results compared to the ML models, the model architectures and trainable parameters are not reported.
[26],2021	Window-based grayscale image, vibration dataset	Self-attentive convolutional neural network (SECNN)	The model shows improved accuracy for an imbalanced dataset. However, the model was validated by generating sample data using GANs due to the unavailability of the actual dataset.
[36], 2022	Constant-Q transform (CQT), AE data	GAN	Although the model has rapid convergence and high accuracy, the model has 0.31 million trainable parameters and is tested with limited environments; the limitations of CQT are its sensitivity to the tuning parameters, computation burden and unsuitability for a large dataset and real-time applications.
[37], 2024	CWT, vibration dataset	CNN-CA (coordinate attention)	The coordinate attention mechanism with CNN shows improved performance. Although the CWT is not required to select a window size as STFT, it is mostly suitable for non-stationary signals and computational complexity is an issue.

presents a brief summary of state-of-art fault diagnosis models and their limitations.

1.2. Contribution Summary

This paper is an extension of [38], where we utilized a lightweight SqueezeNet with the self-attention mechanism. Motivated by the papers [30,31,37], in this extended paper, we have applied three more attention mechanisms, namely CBAM, channel, and spatial, and then compared these with the self-attention mechanism. The contributions of the paper are as follows:

• This paper presents the classification results of the lightweight SqueezeNet with four attention mechanisms: self-attention, CBAM, channel, and spatial. With the additional attention mechanisms, the model is lightweight compared to state-of-the-art architectures, such as CNN, VGG, DenseNet, and ResNet. The trainable parameters of the SqueezeNet architecture with and without attention mechanisms are 0.73~0.82 million, whereas the trainable parameters of the state-of-art architectures are 1.3~138 million [38].
• All the experiments run with texture images of Malfunctioning Industrial Machine Investigation and Inspection (MIMII) and Toy Anomaly Detection in Machine Operating Sounds (ADMOS) audio sensor datasets, which are generated by applying EMD and Gammatone Spectrogram. In [38], it is shown that the EMD–Gammatone spectrogram generates more unique texture patterns for each type of fault compared to the MFCC, Gammatone, and Hilbert–Huang transform. The experimental results show that the EMD–Gammatone spectrogram demonstrates 88.81% accuracy for the SqueezeNet architecture, whereas the other state-of-the-art texture generation techniques exhibited only 62~79.42% accuracy.
• The experimental results show that among the attention mechanisms, the self-attention mechanism outperforms the other three attention mechanisms. It is concluded that the self-attention mechanism can acquire the most effective information from the input vector forwarded by the convolution and pooling layers. The experimental results show that the self-attention mechanism exhibits 97% accuracy for the MIMII and ToyADMOS datasets, whereas the other attention mechanisms exhibited 95~96% accuracy.
• In addition, to explain the classifier results step by step, the t-SNE is applied to visualize the features. We run all the experiments for 200 epochs and report the features for every 100 epochs of the training dataset, and the features of the testing dataset after 200 epochs both for MIMII and ToyADMOS for the SqueezeNet architecture with the self-attention mechanism. The t-SNE results illustrate that the model with a self-attention block almost accurately extracted the distinct features of each fault both for balanced and imbalanced datasets.

The rest of the paper is organized as follows. Section 2 presents a detailed methodology of the proposed classification framework, explaining different attention mechanisms with relevant equations. An analysis of the experimental results is presented in Section 3 considering different performance metrics, such as the accuracy, confusion matrix, training and loss curves, and receiver operating characteristic (ROC) plot. The interpretation of the Self-Attention SqueezeNet Results with t-SNE is presented in Section 4. Finally, the paper is concluded in Section 5, along with the limitations of this research and future directions.

2. Methodology

The fault diagnosis model presented in this paper (as shown in Figure 1) followed the following steps: data preprocessing, texture spectrogram image generation, and the classification of the machine audio faults using a lightweight SqueezeNet followed by attention blocks. The detailed steps associated with the model are presented in Algorithm 1. As an extension of the paper [38], in this paper, we compare the performance of the self-attention SqueezeNet model with three other attention mechanisms: channel, CBAM, and spatial.

The self-attention architecture, as depicted in [38], consists of a self-attention layer preceded by two 2D convolutional layers with three fire modules in between, and a global max pooling layer. The self-attention layer is responsible for computing the logits. The fire module is one of the core components of our model. It comprises three convolutional layers. The first layer is a 1 × 1 convolution that reduces the number of filters to the “squeeze” parameter, followed by two parallel convolutional layers: one 1 × 1 convolution and one 3 × 3 convolution. These layers each produce “expand//2” filters. The outputs of these parallel layers are concatenated to form the final output of the fire module. This design reduces the computational complexity while maintaining rich feature extraction capabilities.

The total trainable parameters of the SqueezeNet architecture is 73,632 and with additional attention blocks, this increases to a maximum of 82,224. The model is lightweight compared to the state-of-the-art architectures, where the trainable parameters of CNN [39,40], DenseNet151 [41], ResNet50 [42], and VGG16 [43] are 1.3 million, 20 million, 23 million, and 138 million, respectively.

Algorithm 1: Steps followed to implement the proposed fault detection model

Step 1: Signal pre-processing and texture image generation   (1) Resize the audio signals into a 1024 sample size.   (2) Select the top 6 IMFs applying the EMD operator.   (3) Regeneration of audio signals by combining the top 6 IMFs.   (4) Apply the gammatone filterbank to generate the texture of combined signals generated in (3) Step 2: Train and test with SqueezeNet architecture followed by an attention block (self-attention, channel, CBAM, spatial). Experimental specifications are as follows,   (1) Input size of 32 × 32   (2) Batch size of 32, number of epochs 200   (3) Activation function used in conv2D and fire module is ‘relu’   (4) Optimizer used Adam (Adaptive Moment Estimation); learning rate is set to ‘5 × 10−4’; loss is set to ‘sparse_categorical_crossentropy’   (5) The early stopping function is used with a patience of 150 Step 3: Features visualization using t-SNE after every 100 epochs for the train set and the test set after 200 epochs

The encoder model begins with an input layer that accepts tensors of the specified input shape. This input is then processed through an initial convolutional layer with a 3 × 3 kernel and 16 filters, utilizing ‘ReLU’ activation and the same padding. Following this, the data are passed through a sequence of three fire modules, which are innovative structures combining 1 × 1 and 3 × 3 convolutions to balance complexity and performance, with the first module configured with 24 squeezes and 48 expand filters, and the subsequent two modules each with 48 squeezes and 96 expand filters. After each of the first two fire modules, a MaxPooling2D layer with a pool size of 2 is applied to reduce the spatial dimensions. After the third fire module, another convolutional layer with a 3 × 3 kernel and 16 filters is applied. The output of this layer is then subjected to global average pooling, reducing the feature map to a single value per filter. This pooled output is processed by a multi-head attention layer with two heads and a key dimension of 64, focusing attention across one axis and outputting the transformed tensor along with attention scores. Finally, the resulting tensor is fed into a dense layer with a softmax activation function to produce the class probabilities.

The following subsections present the channel details, CBAM, and spatial attention mechanisms.

2.1. Channel Attention Mechanism

After passing through convolutional layers, the channel attention produces a multi-channel feature map, $F\in R^{C\times H\times W}$ [44], where C, H, and W represent the number of channels, height, and width of the feature map, respectively. The information of each channel is different. The channel attention uses the relationship between the feature map of each channel to learn a one-dimensional weight, $W_c\in R^{c\times 1\times 1}$, and then multiply it with the respective channel. With the above principle, the channel attention mechanism focuses on more meaningful information. In [9], the channel attention is computed as follows:

$$M_c\left(F\right)=\sigma (\mathrm{M}\mathrm{L}\mathrm{P}\left(\mathrm{A}\mathrm{v}\mathrm{g}\mathrm{P}\mathrm{o}\mathrm{o}\mathrm{l}\left(\mathrm{F}\right)+\mathrm{M}\mathrm{L}\mathrm{P}\left(\mathrm{M}\mathrm{a}\mathrm{x}\mathrm{P}\mathrm{o}\mathrm{o}\mathrm{l}\left(\mathrm{F}\right)\right)\right)=\sigma (W_1(W_0(F_{avg}^C))+W_1(W_0(F_{max}^c)))$$

(1)

where $\sigma$ stands for the sigmoid function, $W_0\in F^{C/r\times C}$, and $W_1\in R^{C\times C/r}$. The MLP weights ($W_0$ and $W_1)$ are shared for inputs and the ‘relu’ activation function is followed by $W_0$. The channel attention module is presented in Figure 2, where both the average pooled and max pooled features are used simultaneously.

2.2. Spatial Attention Mechanism

The spatial attention mechanism [9,45] assigns a higher weight to the more informative features of each frame. Figure 3 shows the spatial attention mechanism, where Max-pooling and Global average pooling are applied to the feature map, and F is applied to interpret the features. Then, a convolution layer with a kernel size of 3 and a filter size of 16, along with the ‘relu’ activation function, is applied to calculate the attention map, $A_s(F)$. The $A_s(F)$ is calculated as [46],

$$A_s(F)=\sigma (F^{3\times 3}([MaxPool(F); GlobalAvgPool(F)])={\sigma {(F}^{3\times 3}([F}_{max}^S;F_{GlobalAvg}^S]))$$

(2)

where $\sigma$ is sigmoid function and $F^{3\times 3}$ is used for a convolutional operation with a filter size of 3 × 3.

2.3. CBAM Attention Mechanism

The CBAM has two sequential sub-modules, namely channel and spatial [9]. Considering an intermediate input feature map, $F\in R^{C\times H\times W}$, CBAM sequentially infers a one-dimensional channel attention map, $W_c\in R^{c\times 1\times 1}$, and a two-dimensional spatial attention map, $M_s\in R^{1\times H\times W}$. The attention processing can be expressed as

$$F^{\prime }=M_c(F)\otimes F \mathrm{and} F^{″}=M_s(F^{\prime })\otimes F^{\prime }$$

(3)

where $\otimes$ stands for element-wise multiplication. $F^{″}$ is the final refined output and, during the multiplication, channel attention values broadcast along with the spatial dimension and vice versa. Figure 4 illustrates the detailed blocks associated with the CBAM mechanism.

3. Experimental Results Analysis

The following subsections present the results of different experiments performed with the lightweight SqueezeNet followed by four attention mechanisms. All the simulation results are collected by running the model for 200 epochs both for MIMII [47] and ToyADMOS [48]. The MIMII dataset consists of eight classes, including normal and abnormal faults in a fan, pump, slider, and valve. In Figure 5 and Figure 6, 0 stands for fan abnormal, 1 for fan normal, 2 for pump abnormal, 3 for pump normal, 4 for slider abnormal, 5 for slider normal, 6 for valve abnormal, and 7 for valve normal. In all cases, normal samples are almost 3 times higher than the abnormal fault samples; details of the number of samples are presented in [38]. There are three types of machines in ToyADMOS, namely toy car, toy conveyor, and toy train, where each machine has normal and abnormal faults. Compared to the MIMII dataset, ToyADMOS is an imbalanced dataset, as the total samples for toy car is 6573, where 250 and 150 samples are for toy train and toy conveyer, respectively. During dataset preparation, a sample with a size of 1024 is taken from each large sample signal. To sift out the high-frequency constituents from the input signal, we selected the top six IMFs for generating a combined signal. Then, we applied the gammatone spectrogram to generate a spectrogram texture image. Then, after resizing the spectrogram image to 32 × 32, the input images were fed to the architectures for training and testing the model. As performance metrics, we utilized precision, recall, F1-score, and confusion matrix. In addition to the different metrics, different visualization curves such as the training and validation accuracy, loss curves, ROC curves, and t-SNE were used to explain the results of the model with four different attention mechanisms.

3.1. Experiment with MIMII Dataset

The performance of various modified SqueezeNet models was evaluated on the MIMII dataset.

Table 2. Classification report of the tested models for the MIMII dataset.

	Trainable Parameters	Accuracy	F1-Score
SqueezeNet without attention	73,632	96%	0.96
SqueezeNet with Self-attention	82,224	97%	0.97
SqueezeNet with Channel attention	73,796	96%	0.96
SqueezeNet with Spatial attention	73,730	96%	0.96
SqueezeNet with CBAM attention	73,894	96%	0.96

presents the results of these models, including the number of trainable parameters, accuracy, and f1-score for each model.

The SqueezeNet model integrated with self-attention exhibited a superior performance compared to the other models. It achieved the highest accuracy of 97% and an F1-score of 0.97, indicating that the self-attention mechanism enhances the model’s ability to identify faults accurately. The other models, which incorporated channel attention, spatial attention, and CBAM attention, all achieved an accuracy of 96% and an F1-score of 0.96, demonstrating their effectiveness but not matching the performance of the self-attention model.

Furthermore, as shown in Figure 7, the validation loss and accuracy curves for the self-attention SqueezeNet model were more stable compared to those of the other models. This stability suggests that the self-attention mechanism contributes to a more consistent training process, reducing fluctuations in model performance and leading to more reliable fault diagnosis results. Figure 8 shows the ROC curve of the self-attention SqueezeNet model, which were almost similar to the other attention mechanisms.

3.2. Experiment with ToyADMOS

The performance of different modified SqueezeNet models was assessed using the ToyADMOS dataset.

Table 3. Details of the classification report for the models tested using the ToyADMOS dataset.

	Accuracy	F1-Score
SqueezeNet without attention	96%	0.96
SqueezeNet with Self-attention	97%	0.97
SqueezeNet with Channel attention	97%	0.97
SqueezeNet with Spatial attention	96%	0.96
SqueezeNet with CBAM attention	95%	0.95

summarizes the results, highlighting the number of trainable parameters, accuracy, and F1-score for each model. In Figure 9 and Figure 10, 0 stands for car abnormal, 1 for car normal, 2 for conveyer abnormal, 3 for conveyer normal, 4 for train abnormal, and 5 for train normal.

As depicted in Figure 9, both the SqueezeNet model with self-attention and the one with channel attention achieved an accuracy of 97% and an F1-score of 0.97 on the ToyADMOS dataset, which were the highest values. This highlights that the self-attention and channel attention mechanisms both significantly enhance the model’s fault identification capabilities. The models with spatial attention and CBAM attention demonstrated a strong performance as well, with accuracies of 96% and 95%, and F1-scores of 0.96 and 0.95, respectively. These results confirm the effectiveness of these attention mechanisms, though they fell short of the self-attention and channel attention models’ performance.

Additionally, the validation loss and accuracy curves (as shown in Figure 11) for the self-attention SqueezeNet model were more stable compared to the other models. This indicates that the self-attention mechanism contributes to a more consistent training process, reducing performance fluctuations and resulting in more reliable fault diagnosis outcomes.

However, it is important to note that the ToyADMOS dataset exhibited extreme class imbalance, posing a significant challenge for the models, particularly in handling minority classes. The models, including those with self-attention and channel attention, struggled considerably to accurately identify faults in these underrepresented classes, impacting their overall performance and highlighting an area for further improvement in future research. Like the MIMII dataset, the ROC curve of the self-attention SqueezeNet model for ToyADMOS was almost similar to the other attention mechanisms, as shown in Figure 12.

4. Interpretation of Self-Attention SqueezeNet Results with T-SNE

Figure 13 and Figure 14 exhibit t-SNE plots of the self-attention SqueezeNet logits using the MIMII and ToyADMOS dataset, respectively. The plots were generated at 100-epoch intervals throughout the 200-epoch training session. The figures also include two additional plots: one representing the t-SNE plot with no training (epoch 0) and the other depicting the t-SNE plot on the test set of MIMII and ToyADMOS.

These visualizations provide insights into how the model’s feature space evolves during training and its effectiveness in differentiating between classes. The progression of the t-SNE plots over the epochs highlights the model’s ability to learn and separate different fault types more distinctly as training progresses. The comparison between the training and test set t-SNE plots further demonstrates the model’s generalization capability.

The t-SNE plots in Figure 13 demonstrate a clear separation of clusters representing different fault types. This indicates that the self-attention SqueezeNet model effectively learned to distinguish between various fault categories, with the separation becoming more distinct as training progressed. These well-defined clusters in the feature space highlight the model’s robust capacity for fault identification and classification.

In contrast to Figure 13, Figure 14 shows the t-SNE plots of the self-attention SqueezeNet logits using the ToyADMOS dataset. Here, the clusters representing different fault types are less clearly separated. This observation is primarily attributed to the class imbalance present in the ToyADMOS dataset, where certain fault types are underrepresented compared to others. This imbalance poses a challenge for the model in learning distinct representations for minority classes, resulting in fewer distinct clusters in the feature space visualization.

Despite these challenges, the t-SNE plots in Figure 14 still provide valuable insights into the model’s learning process and its attempt to differentiate between fault types. Addressing class imbalance through techniques like data augmentation or class weighting could potentially improve the model’s performance on datasets such as ToyADMOS, enabling the clearer separation of clusters and enhancing the overall fault classification accuracy.

5. Conclusions

While the model is designed to be computationally efficient, the incorporation of multi-head attention mechanisms still introduces a non-negligible computational overhead. This could be a constraint when deploying the model on resource-limited portable devices. The further optimization and exploration of lightweight attention mechanisms could mitigate this issue. Moreover, the ToyADMOS dataset’s class imbalance significantly impacted the model’s performance, particularly for minority classes. Although the model achieved high accuracy overall, its ability to accurately identify faults in underrepresented classes was diminished.

In conclusion, this paper presented a comparative analysis of four attention mechanisms: self-attention, channel, spatial, and CBAM. All the experiments were run using two public datasets: MIMII and ToyADMOS. Among the attention mechanisms, the self-attention mechanism accurately extracts more useful information from the fault datasets. The experimental results show that the additional self-attention blocks do not increase the number of trainable parameters in the overall architecture and that the self-attention mechanism demonstrates a 0.97 F1-score, whereas the other three mechanisms show a 0.96 F1 score for the MIMII dataset. For the ToyADMOS dataset, self-attention demonstrates a 0.97 F1-score, whereas the channel, spatial, and CBAM exhibit F1-scores of 0.97, 0.96, and 0.95, respectively. The results demonstrate that the self-attention SqueezeNet model achieved the highest accuracy and F1-scores on both datasets, indicating its superior fault identification capabilities.

The limitations of this study are that the model is tested with an imbalanced dataset and that the computational complexity of it is only determined with trainable parameters. In the future, data imbalance issues could be handled with data augmentation techniques and the model’s computational complexity could be addressed using optimal feature selection with explainable AI techniques.