Multiclass Anomaly Detection of Bridge Monitoring Data with Data Migration between Different Bridges for Balancing Data

Abstract

Structural health inspection systems are widely used to manage and maintain infrastructure that involves massive sensor devices. However, these sensors receive the natural environment or external factors in the long-term exposure to the outdoor environment, resulting in the failure of the sensors, which causes multiple categories of abnormal data in the collected data. The data often is unbalanced due to the random nature of failures. This unbalanced anomaly data poses a major challenge to existing anomaly detection methods and will affect the effectiveness of the information provided by the structural health monitoring system. In the paper, a data migration method is proposed to migrate bridge data to the target bridge dataset for expansion so that the number of images of different categories in the target bridge dataset increases. This method can be divided into three steps: firstly, to classify the data; secondly, to determine the suitability of the data and to construct the dataset; and finally, to train the data. The comparative validation is used to compare the training performance of the dataset using data migration with the dataset only using the target bridge to analyze the abnormal data identification in each category. In the experiment, the recall of some categories of data reached a significant increase of more than 30%, achieving better identification of various categories of abnormal data. Adopting the method of data migration between different bridges can solve the impact of imbalanced data and improve the recognition performance of categories with fewer images.

1. Introduction

Over time, worldwide civil infrastructure is aging as natural disasters because of increased population and operational load [1,2,3]. Structural health monitoring (SHM) technology is gradually emerging as an important means to monitor the “health” of engineering objects [4,5]. The use of intelligent sensing systems for real-time monitoring, dynamic management, and trend research of engineering structures [6,7]. With the development in the last decade, structural health inspection technology has been widely used in the maintenance and management of civil infrastructure such as bridges, tunnels, and buildings [8,9].

Sensing the important information of the building structure’s behavioral response to external environmental erosion, extreme load effects, and peripheral disturbance intrusion in real time by forming a monitoring network with various sensors pre-buried or attached inside or on the surface of the monitored object [10,11]. It can be reinforced and maintained in advance to avoid unpredictable dangers which cause a significant loss of life and property to the people and prolong the life of civil infrastructure when abnormal information is found in the research and evaluation that buildings are broken and damaged [12,13]. Structural health inspection technology can make intelligent assessments and decisions on the health condition of civil infrastructure, such as damage destruction, performance degradation, and operation efficiency [14,15]. It provides a scientific basis for the maintenance and management of engineering structures, life prediction, and security usage [16,17]. The sensors installed on the structure collect each physical quantity to be sensed in real time, timed, triggered, or mixed mode, gathering a large amount of information over a long period of time. However, there is a lot of abnormal information contained in them due to their leakage in the outdoor environment during the structural health detection systems processes, which are easily affected by the surrounding environment, external forces, or its own factors such as sensor failure and abnormal transmission [18,19]. For example, the signals collected by the sensors may be mixed with various noises due to the interference of the surrounding environmental factors, which makes it difficult to extract and analyze their valid information accurately. This cluttered information can interfere with normal analysis and may lead to incorrect judgments about the health of the engineering structure, which causes a lot of waste of human and material resources [20]. Anomaly detection (AD) techniques are required so that only high-quality data is available for analysis of the state of the structure [21,22].

AD is an important process for SHM to cleanse data before implementing any diagnostic algorithms, which can analyze and clean the data to obtain high-quality data for effective analysis of structural health conditions. AD method based on convolutional neural network (CNN) is developed with the development of machine learning and deep learning. The CNN is a type of neural network designed to process data with a grid-like structure [23,24]. It enables feature extraction of things through the role of CNN species convolutional layer, pooling layer, and fully connected layer [25]. Additionally, then the features are used to classify, identify, predict, or make decisions about the thing [26]. One-dimensional CNN is designed to extract from the input signal for the detection of acceleration data from SHM systems of large-span bridge in China directly. The results show that the method can detect anomalies with high accuracy in the anomaly detection stage [27]. A new CNN framework is proposed. It combines the advantages of CNN and statistical features and can achieve faster and more accurate data anomaly detection than using CNN alone [28]. A method converts time series signals into images to visualize the data and categorizes them into six categories according to the different features of the images. Then, uses the trained neural network to recognize the images and realize the classification of the target data [29]. The method converted the time series data into a 3-channel image consisting of time, spectral, and probability density functions. Then, used the designed CNN to recognize the images, which achieved high overall accuracy and recall [30]. Two-dimensional CNN is used for the recognition of time-frequency domain images. The frequency domain response is obtained from the time domain data by fast Fourier variation, and the approach is validated for a large span bridge in China with real data and good results [31].

When using CNN to solve problems, when one or more of the predicted categories has a very small sample size, it may face the problem of unbalanced categories. The unbalanced distribution of samples will result in categories with small sample sizes containing too few features, and it is difficult to extract patterns from them, and even if a classification model is obtained, it is prone to the problem of overfitting due to over-reliance on a limited number of data samples. When the model is applied to new data, the accuracy and robustness of the model will be poor, and the optimal results cannot be obtained in real time because the model never fully examines the implied classes. Based on this, this paper proposes a data migration method to migrate other bridge data to the target bridge data set. The method mainly consists of three parts. Firstly, the data are visualized and classified according to the image features. Secondly, determine whether the time domain images of other bridge data can be applied to the target bridge, and then select the data to add to the target bridge training set. Finally, the selected pre-trained model is employed in a circularly new training set to obtain an abnormal data detection model with better performance. The model can complete the classification of all categories of abnormal data to solve the problem that the target bridge cannot accurately identify all categories of data due to the lack of data imbalance of a certain category of data.

2. Abnormal Data Feature Identification

Firstly, time domain analysis is used to process the data. The data is visualized to transform time series data into images for better classification based on their characteristics of them. The horizontal coordinate of each image is the time, and the vertical coordinate is the acceleration. Each image occupies the duration of one hour of the original data, and the data characteristics of one hour of acceleration are reflected in this graph roughly. The characteristics of different categories of monitoring data are analyzed by the trend of acceleration amplitude in images over time and space. Each category of data will be divided into six categories: normal, drift, gain, noise, missing, and outlier. Figure 1 shows an example of each category of image.

The normal data images have acceleration amplitudes that fluctuate in an orderly manner above and below the image centerline with significant fluctuations. The drift data image has a tendency for the data to deviate from the centerline and shows a tendency to tilt or fluctuate above and below the centerline. The local gain data images show a tendency for a sudden increase in acceleration amplitude in some time periods, and their amplitude is much higher than the data in other time periods. The noise data images whose acceleration amplitude tends to be consistent throughout the time period without significant fluctuations. The missing data image has some data missing; the acceleration amplitude at the missing point is zero. Additionally, a straight line is shown in the image, or the whole image is blank with all data missing. The outlier data image has a sudden increase in acceleration amplitude at a certain moment, and one or more isolated data points are presented in the image as well as the overall features are significantly reduced.

3. Construction of Convolutional Neural Network Model Based on Transfer Learning

3.1. Transfer Learning

Transfer learning is a machine learning method whose main purpose is to extract knowledge from one or more source domains to apply the knowledge to the target domain. Additionally, deep transfer learning has improved accuracy compared to the traditional random initialization of weights. The effect of deep neural networks in task classification is enhanced by sharing the weights and features through the migration of the network layers. It avoids training a neural network from scratch for the purpose of accelerating network optimization. In this paper, transfer learning is adopted to select the CNN, which has performed massive image classification tasks on the ImageNet training set as the initial model. Additionally, use the transfer learning model to train the bridge detection data so that the model has better generalization and accuracy to achieve the classification of six categories of bridge monitoring data.

3.2. Pre-Trained Network Model Selection

Based on the transfer learning, the classical Resnet 50 network model in the pre-trained network is selected as the base model in this study. The Resent 50 model has a unique residual structure compared to other pre-trained models. Due to the residual structure, the Resnet is deeper. It has the advantage of stronger feature extraction and recognition capabilities. Considering obtaining a better recognition accuracy, Resnet 50 was selected for multicategory abnormal bridge data. Then, Resnet 50 is fine-tuned to adapt it to the classification and recognition tasks of the bridge acceleration dataset. Figure 2 shows the process diagram of the Resnet 50 model for input image recognition. The model first converts the image size to 3 × 224 × 224, performs initial feature extraction through the convolution layer, and reduces the size of the parameter matrix through the pooling layer to remove redundant information and compress the features. Then, the image features are extracted layer by layer through 16 bottleneck residual blocks, and the pooling layer is downscaled to reduce the number of parameters. Finally, the input images are classified into six classes by the fully connected layer and softmax function. In image recognition, its ability to extract features is stronger, the expression ability will be stronger, and the classification performance will be better theoretically when the number of layers of the deep learning network is deeper. However, the classification performance will not improve when the CNN network reaches a certain depth and deepens again. Rather, it causes the network to converge more slowly, and the accuracy rate decreases with it.

Residual blocks provide shortcut links between layers compared to traditional CNN in Resnet 50. It solves the problem that the training and verification error rate increases as the network deepens, the gradient explosion or gradient disappearance, and improves the efficiency and performance of the network. The structure of the residual block is shown in Figure 3.

In a residual network, the input samples are denoted as $H\left(x\right)$, with the output features denoted as $H\left(x\right)$. Additionally, the mapping of the input to the output is learned directly in the ordinary network. Due to the presence of shortcut connections, the residual network learns the difference between $H\left(x\right)$ and $x$, which is called the residual that is $F\left(x\right)=H\left(x\right)-x$. When $F\left(x\right)$ converges on 0, the output is approximated as $H\left(x\right)=x$. Additionally, the residual network achieves constant mapping. Learning $F\left(x\right)=0$ is easier than learning $H\left(x\right)=x$, and the network achieves faster convergence.

4. Data Validation

4.1. Dataset Construction

Before training the data images, it is necessary to manually select the data and select a small portion of the data to build the dataset. In order to better verify the necessity and effectiveness of data migration between different bridges, A, B, and C, three different bridges data were cumulatively selected for the construction of a training set, validation set, and test set. The data of A bridge and C bridge were selected to build expanded data sets for data migration. Then, the data sets of input CNN are constructed for the target B bridge and A bridge. The specific data volumes for each category of the A bridge are shown in

Table 1. The specific data volumes for each category of the A bridge dataset.

Data Category	Number of Images
normal	14,613
drift	1802
local gain	1409
noise	1265
missing	1850
outlier	153

, the specific data volumes for each category of the B bridge are shown in

Table 2. The specific data volumes for each category of the B bridge dataset.

Data Category	Number of Images
normal	1463
drift	4621
local gain	244
noise	21,293
missing	10,503
outlier	1524

, and the specific data volumes for each category of the C bridge are shown in

Table 3. The specific data volumes for each category of the C bridge dataset.

Data Category	Number of Images
normal	21,510
drift	162
local gain	21
noise	45
missing	202
outlier	398

.

4.2. Evaluation Indicators

The confusion matrix is used to visualize the performance of the classification model. The predicted categories of data images in the confusion matrix correspond to the actual categories, and the data reflected in each cell of the matrix can show how each category of bridge anomaly data is classified by the model. The evaluation metrics Accuracy, Precision, and Recall of the confusion matrix are defined as shown in Equations (1)–(3), respectively [32].

$$\mathrm{Accuracy}=\frac{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}$$

(1)

$$\mathrm{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$$

(2)

$$\mathrm{Recall}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$

(3)

True positive (TP) indicates that the actual value is the same as the predicted value and correctly classifies the target category samples into the belonging categories; false positive (FP) indicates that the actual value is negative, but the model predicts a positive value that misclassifies samples from other categories into the target category; true negative (TN) indicates that the actual and predicted values are the same, which means that the samples from other categories are correctly classified into the other categories to which they belong; false negative (FN) indicates that the actual value is positive but is predicted by the model to be the negative value, meaning that the target category samples are misclassified into other categories. Precision is the probability of the samples which the model considers correct and are indeed correct as a percentage of all samples considered correct by the model. Recall is the probability for the samples that the model considers correct and are indeed correct as a percentage of all samples which are actually correct. Accuracy is the percentage of correctly predicted results in the total sample.

Accuracy and Recall will be used as the main evaluation analysis metrics to evaluate the effectiveness of data migration when data is not balanced among these metrics. Accuracy reflects the overall performance of the model. Recall is more realistic for bridge data anomaly detection than Precision, which can find each category of abnormal data as much as possible.

4.3. Analysis of Results

In order to verify the effectiveness and necessity of data migration between different bridges, the recognition effect of the model trained by using the target bridge training set with extended data sets is compared with the recognition effect of the model trained by all the target bridge data training set, so that the Accuracy of the proposed problem can be verified.

To illustrate the quality of the expanded dataset of A bridge, firstly, the expanded dataset is trained using the Resnet 50 model alone, and the dataset will be divided into a training set and a validation set according to the ratio of 8:2.

Figure 4 shows the accuracy and loss function curves obtained during the training process, where the horizontal coordinate is the number of iterations of model training and the vertical coordinate is the Accuracy. From Figure 4, we can see that the Accuracy is high when the initial iteration of the convolutional neural network is performed, which means that the training model of this dataset has good classification ability for six categories of data. The A bridge dataset can be used as an expanded training set.

The data volume of each category in the B bridge training dataset is shown in Table 2, from which it can be seen that the local gain data volume is seriously low, and the data is unbalanced. The Resent 50 model is used to train the data, and the hyperparameters are manually adjusted and set empirically. MaxEpoch is set to 1, InitialLearnRate is set to 0.001, Optimizer is set to SGD, and MiniBatchSize is set to 32. Figure 5a shows the confusion matrix obtained by training the model using the B bridge dataset. Then, we used the data for the other month of B bridge for testing. From Figure 5a, we can see that the model has a seriously low recognition rate for normal, drift, and local gain data, compared to noise, missing, and outlier data. The model cannot identify the above three categories of data effectively. The recognition accuracy of the model in CNN has a certain relationship with the image data. Theoretically, the model learns the features of the data more fully with more images, and the higher the recognition accuracy of the model for the category when the test is finally conducted. The reason for the low recognition accuracy of the local gain category data in Figure 5a can be attributed to the low amount of training data for the above categories and the seriously imbalanced data.

To illustrate the effectiveness of data migration for comparison and validation, a new dataset was constructed using the A bridge expansion dataset plus the B bridge training dataset and then retrained using the network model. The confusion matrix was then obtained via the same test set of the B bridge. The recognition results are shown in the Accuracy of the model for the six categories of data is 93.1% from Figure 5b, which is good recognition ability. Additionally, normal data, drift, and local gain data recognition Recall improved significantly, 30%, 8%, and 22.5%, respectively. Noise, missing, and outlier data recognition accuracy fluctuates normally with no significant decrease. The migration of A bridge dataset is equivalent to the expansion of B bridge data, which is introduced to make it realize the effective classification of B bridge data and alleviate the problem of not being able to realize the accurate classification of six categories of data due to the unbalanced B bridge dataset.

To illustrate the generality of this method, the expanded dataset is changed from A bridge to C bridge. C bridge expanded dataset is used to train the Resnet 50 model, and the training will divide the data set into training and validation sets in the ratio of 8:2. Figure 6 shows the accuracy curves obtained during the training process. From Figure 6, which can be seen that the fitting effect of the training and validation sets of the modified model is good, and the recognition accuracy of the validation set is over 90%. The training set is trained to obtain a model with high recognition accuracy, and this C bridge dataset can be used as an expanded dataset. From Table 1, the training data of A bridge is also imbalanced. Figure 7a shows the confusion matrix obtained by training using the training set of A bridge and then testing using the test set of A bridge. Figure 7b shows the confusion matrix obtained by using the expanded dataset of C bridge plus the dataset of A bridge and then testing with the same test set of A bridge. From the comparison of the two figures, we can see that the Recall of drift, local gain, noise, missing, and outlier is improved by 2.1%, 3.7%, 1.3%, 0.4%, and 34.6%, respectively. The normal data recognition accuracy fluctuates within the normal range, but there is no significant decline in adopting data migration; the overall recognition rate of the model for different categories of data can be improved, and the low recognition accuracy of certain categories caused by unbalanced data is alleviated. By using the C dataset as the expanded dataset, the same effect as the A bridge expanded dataset can be achieved, which illustrates the generality of this method. The bridge dataset can be used as an expanded training set if it satisfies the requirement of having good classification capability for its own bridge data.

The effectiveness of data migration was verified by comparative validation, and two working conditions were set up separately. After data expansion of B bridge using A bridge, the normal data Recall is improved by 30%, and other categories of data also have a small improvement after data expansion of A bridge using C bridge data. The Recall of jump point data is improved by 34.6%, and other categories of data also have a small improvement. It illustrates that the adoption of migration learning effectively increases the data features of the dataset samples, improves the recognition accuracy of the convolutional neural network for different categories of data, and enhances its ability to detect abnormal data.

5. Conclusions

There are many categories of abnormal data in the collected data due to the failure of sensors or other factors for SHM, and the data in each category is often imbalanced. To better facilitate the anomaly detection of SHM data and solve the problem of having low accuracy in identifying samples with a small number of images because of the unbalanced dataset, this paper proposes the data migration method for identifying SHM abnormal data.

The process of data migration reveals that there are some similarities in the image features of different bridge data, and other bridge data can be migrated to the target dataset. The models trained on the target bridge dataset with data migration perform better overall than the models trained without data migration for different categories of abnormal data. In this paper, the test conditions showed a 30% improvement in some categories. The recognition accuracy of the categories with fewer images has been improved.

The bridge data used for data migration can be selected from the dataset of other bridges. So long as the model trained on this dataset has good recognition ability for its own bridge data, it can be migrated and expanded to the dataset constructed by the target bridge, and this method is generalized.

The proposed method in this paper also has some limitations. When data migration is performed, a manual judgment of the suitability of image features between bridges is required. This process is not automated.