Trident: A Deep Learning Framework for High-Resolution Bridge Vibration Monitoring

Abstract

Bridges are the essential components in lifeline transportation systems, and their safe operation is of great importance. Information on structural damage could assist in timely repairs and reduce downtime. With the latest advancements in sensing technology, collecting vibration data from bridges has become more accessible. However, effective vibration processing is still a challenge, given the high dimensionality and massive size of vibration data. Existing studies have shown that machine/deep learning techniques can be valuable tools for this task. However, the learning and computational capacities of these models are challenged in the presence of large sensor arrays. We propose Trident as a novel deep learning framework that enables automatic damage feature extraction by simultaneously learning from temporal and three-dimensional (3D) spatial variations of 6D input data in instrumented bridges. Trident is equipped with 3 ConvLSTM3D branches to achieve this goal. A 3D steel truss bridge subject to dynamic traffic loads is monitored for its vibrations to evaluate Trident’s robustness in finding damaged elements. A damage dataset of 52,800 vehicle passing simulations is generated leveraging a database of 528 passenger vehicles in the United States, obtained from the National Highway and Traffic Safety Administration. Bayesian optimization is utilized to tune the model’s hyperparameters, achieving a test Node Average Geometric Mean Accuracy of 86%. This level of performance is promising given the high dimensionality and complexities of the output space in vibration-based monitoring. Trident’s concept can be extended to other vibration monitoring tasks with different time series data and damage labeling strategies.

1. Introduction

Structural Health Monitoring (SHM) is the practice of implementing a damage identification strategy for civil, mechanical, and aerospace infrastructure [1]. In buildings and bridge structures, classical methods often rely on human expertise and operation as an essential part of the damage diagnosis process, limiting the potential for continuous monitoring. Identifying damage at early stages could reduce repair costs and minimize downtime of bridges as an essential part of the transportation system. There are more than 600,000 bridges across the United States, among which 7.5% are in poor condition. There is great value in this automation, considering the scale of civil infrastructure development.

While promising, this automation is challenging using traditional software development based on explicit step-by-step instructions for a computer program. Artificial Intelligence (AI) provides another alternative by learning abstract features between the input and output space using a large dataset of real-world observations and a model with sufficient generalization capacity (e.g., a deep neural network). In recent years, there has been a lot of focus on developing automated SHM frameworks leveraging the latest advances in AI and, more specifically, deep learning. An extensive number of research studies have investigated the potential application of deep learning to process raw camera footage in various tasks. Detecting structural components and localizing different types of structural damage through bounding boxes or pixel-wise semantic segmentation in global to local scales are examples of efforts in automating visual inspections [2,3,4,5,6]. A detailed review of deep learning applications in vision-based SHM can be found in Dong and Catbas [7] and Spencer et al. [8]. There has been substantial progress in this domain mainly because neural network architectures initially proposed by deep learning research can benefit a variety of vision tasks, including images from building façades or highway bridge inspections. From a more technical point of view, the image dataset’s input and output tensor structure does not change substantially despite the significant difference in data distributions. To this end, it is shown that transfer learning can also help to fine-tune pretrained models to perform well for structural engineering purposes [9].

Vibration-based health monitoring is an alternative to visual inspections that involves processing structural responses, like accelerations, into a measure of structural damage. Several works in the literature have utilized machine learning to process vibration data into information about the damage. Classical machine learning algorithms, including but not limited to K-nearest neighbors, tree-based methods, clustering techniques, multi-layer perceptrons, and support vector machines, have been studied for their robustness for damage diagnosis [10,11,12,13,14,15,16]. Most of these traditional ML models rely on highly processed input features, such as peak response or spectrum values for vibration records (e.g., due to seismic events). Obtaining these manually engineered input features is often associated with significant loss of information because time-series information is processed into highly compact vector representation with a few scalar indices. Sajedi and Liang [17] proposed the concept of Semantic Damage Segmentation (SDS) and a grid environment to process a large array of cumulative intensity features. The authors use a Fully Convolutional Neural Network (CNN) to predict the location and severity of damage in all the structural nodes of a 10-story-10-bay frame. While the SDS concept proposes an efficient method for processing features from a dense array of sensors and mapping them to a large grid of output damage data, extensive feature preprocessing is still essential. Otherwise, the computational costs of processing raw vibrations from a dense network of accelerometers are restrictive.

Some studies use time-series data with different tricks to obtain a measure of structural damage. For example, Abdeljaber et al. [18] used several independent 1D CNNs for each acceleration node, and Khodabandehlou et al. [19] stacked several sequences and reshaped them into an image-like tensor shape structure as input to a 2D CNN architecture. Eltouny and Liang [20] further extended the concept of grid environment in SDS by proposing an architecture named SCAN that comprises a sequence of a spatial encoder, a temporal encoder, and a spatial decoder to process raw vibrations without manual feature engineering. SCAN was validated on a 2D RC structure case study leveraging unsupervised learning. Xu et al. [21] proposed Long-Short Term Memory Neural Networks to learn from the sequential features of a seismic signal for regional damage assessments. Sajedi and Liang [22] recently proposed Mel Filter Banks (MFBs) as an alternative signal preprocessing technique that allows deep learning architecture to learn from time and frequency variations for bridge column drift regression [23].

Large-scale vibration-based health monitoring is challenged by how to deal with high-dimensional data. Sensors are often placed on a three-dimensional (3D) structure, and each physical node with sensors might contain several sensor channels, such as accelerations in the three principal directions. To the best of the authors’ knowledge, the existing deep learning frameworks in the literature cannot process vibration data without significant manipulations in the vibration input tensor for dimensionality reduction. Furthermore, the structural damage output ideally needs to be represented by maintaining separate dimensions in the 3D physical space, which is missing from the earlier studies. The research presented in this paper proposes Trident, a 3-branched neural network architecture equipped with hybrid recurrent and convolutional layers known as ConvLSTM3D that enable direct feature extraction from 6D vibration input tensors. Trident can also use different network heads designed for various damage prediction tasks, including 3D SDS.

The remainder of this paper is organized as follows. First, different components of the deep learning architecture and their purpose are discussed. The following section proposes details on the vehicle passing damage simulations. Subsequently, the truss bridge modeling and the dataset generation process are provided. The next section is dedicated to tuning Trident’s hyperparameters and documenting the performance metrics, followed by a conclusion.

2. Trident

CNNs and recurrent neural networks (RNNs) have existed for several decades [24,25]. CNNs are one of the backbones of deep learning models developed for computer vision, and RNNs played a significant role in natural language processing [26]. The convolution operator helps to extract features with great computational efficiency compared with fully connected neural network layers. This is a critical advantage for problems that deal with high-dimensional input-output spaces, such as high-resolution images and, in the case of SHM, structures that can be modeled inside a large 3D grid of nodes [27]. Despite this advantage, CNNs may not be effective in learning temporal dependencies in sequential data like time series. Variants of RNNs are common in such circumstances, and have a long history of applications in natural language processing [28] and automatic speech recognition [29]. The disadvantage of RNNs is that their ability to process very long sequences is limited. Furthermore, handling multiple time series using a single model with end-to-end training could be highly restricted by the availability of computational resources. An example of this phenomenon in the SHM domain is monitoring acceleration histories from a dense array of sensors installed on a building or a bridge.

Shi et al. [30] propose the ConvLSTM operator that treats the input data as a spatiotemporal sequence. In simple words, this architecture marries the computational advantages of CNNs with RNN’s capabilities in learning from sequential data. The authors replace fully connected layers in a traditional Long short-term memory (LSTM) cell with convolution operators that assist in input-to-state and state-to-state transitions. It is critical to identify the dimensionality of the vibration input tensor to understand how the ConvLSTM cells operate in Trident. Each node in a structure can be assumed as a part of a 3D grid. This grid is essential as it helps the deep learning model receive an input tensor sensitive to nodes’ relative X, Y, and Z coordinates. The original implementation is developed for precipitation nowcasting and utilizes a 2D geometric grid and, therefore, leverages ConvLSTM2D [30]. However, solid objects, such as structures, are 3D, which justifies the need for ConvLSTM3D cells.

In addition to the physical geometry, each node can contain several channels that record different data streams. For example, a triaxial accelerometer records vibration response in three directions. Moreover, SHM might involve other response types, such as strains which could add to the total number of channels (n_c). In this study, inspired by Sajedi and Liang [22], Mel Filter Banks (MFBs) are extracted from acceleration time series to capture variations in time and frequency domains. The first dimension (horizontal axis) of an MFB tensor corresponds to overlapping time windows, and the second dimension contains filter values that indicate the amount of energy stored in different frequency ranges of each time window. With that said, n_t time frames and n_f filter banks add two dimensions to the input tensor. As a result, a structure’s vibration input data can be considered a six-dimensional (6D) tensor of shape (n_x, n_y, n_z, n_c, n_t, n_f). We only consider acceleration response for sensor channels (n_c = 3) because this type of sensing is more common for vibration-based civil infrastructure monitoring. This concept can be extended to scenarios where other types of sensors (e.g., strain gauges) are utilized. For this reason, Trident (as its name implies) is an architecture with three branches.

Each branch receives a 5D tensor of shape (n_x, n_y, n_z, n_t, n_f), which fits a ConvLSTM3D cell that enables learning from temporal variations of vibrations in a 3D space. The main equations of ConvLSTM3D tensor calculations are expressed in the following [30]:

$$i_t=\sigma (W_{Xi}{X}_t+W_{Hi}{H}_{t-1}+W_{ci}\odot c_{t-1}+b_i)$$

(1)

$$f_t=\sigma (W_{Xf}{X}_t+W_{Hf}{H}_{t-1}+W_{cf}\odot c_{t-1}+b_f)$$

(2)

$$g_t=\tanh (W_{Xc}{X}_t+W_{Hc}{H}_{t-1}+b_c)$$

(3)

$$c_t=f_t\odot c_{t-1}+i_t\odot g_t$$

(4)

$$o_t=\sigma (W_{Xo}{X}_t+W_{Ho}{H}_{t-1}+W_{co}\odot c_t+b_o)$$

(5)

$$H_t=o_t\odot \tanh (c_t)$$

(6)

where X_t denotes the MFB features for each time window (X_t) in the 3D structural grid and H_t tensors are hidden states. σ represents the sigmoid function, and i, f, o, and c correspond to the input gate, forget gate, cell, and cell activation vectors [31]. The weight tensors (W) and biases (b) with indices from X, H, i, f, o, and c are learned through backpropagation. The notation $\odot$ refers to element-wise matrix multiplication. Details of this implementation are further illustrated in Figure 1.

Equipped with the ConvLSTM3D cells, the trident architecture can be assembled. The last hidden states from all branches are concatenated and fed to a series of 3D convolution layers. Trident’s head can be customized depending on the damage labeling strategy. In this study, we have assumed a binary 3D output tensor where nodes with damaged elements will have a value of 1 and 0 otherwise. Given the similarity of this strategy to the pixel-wise semantic segmentation task in deep learning, we will refer to this process as 3D Semantic Damage Segmentation (SDS) which is shown in Figure 2.

3. Vehicle Passing Simulations

A series of finite element numerical simulations are considered to monitor bridge vibrations due to vehicle passings and identify the location of structural damage. These simulations will be used to construct a large dataset explained in the next section and help evaluate Trident’s robustness. Several studies in the SHM literature model the vehicle-bridge interaction by considering the vehicle’s suspension system as a multi-degree of freedom system [32,33,34]. For example, Eshkevari et al. [35] leverage vibration crowdsourcing for bridge modal identification using moving accelerometers on a bridge. This paper takes a different approach and assumes the accelerometers are placed on the bridge and simulate excitations due to passing vehicles and damage in different elements. The numerical simulations are based on the following assumptions: (a) The vehicle moves along a straight centerline; (b) The vehicle maintains constant speed during the passage; (c) Vibrations are recorded for a single vehicle on the bridge; (d) Passenger vehicles with two axles are considered. The axle loads imposed on a bridge can be modeled as a dynamic vertical load affected by the vehicle’s speed [35]. The moving load will have a triangular intensity pattern. However, a road’s surface is not perfectly smooth, and the intensity of vertical loads due to the static weight on each axle depends on deck imperfections [36,37]. This phenomenon can cause oscillations in the imposed vertical loads that are difficult to model due to several sources of randomness. Inspired by Hwang and Nowak [38], we add a random gaussian noise to simulate these effects. The noise is added independently for each structural node in the bridge model and in each vehicle passage simulation. There are two parameters to determine the distortion in load pattern, which are the standard deviation of load intensity factor (${\sigma }_n$) and the noise time step (dt_n). The simulation is recorded with a smaller time-step compared with noise and the load intensity is linearly interpolated for response history analyses. The load intensity factor is also a function of the vehicle’s speed (v). This loading strategy is illustrated in Figure 3.

4. Case Study Dataset

4.1. Bridge Model and Damage Simulations

A quarter-scale steel truss bridge model is monitored for vibrations caused by vehicle passage. The bridge model was initially designed and tested for a study on seismic isolation systems by Warn [39]. The truss geometry is 11.9 × 1.2 × 1.5 m (39 × 4 × 5 ft), and it is comprised of W-shaped chords, posts, transverse beams, and a series of diagonal and X-braces as shown in Figure 4. The estimated model weight is 89 kN (20 kips), and ASTM A36 steel material is used. For each vehicle passing simulation, a random number of elements between [0, 10] are sampled and their stiffness is set to 1% of the original value to simulate damage. Truss elements between structural nodes are considered as potential damage locations. Trident can be calibrated to perform regression by outputting the ratio of damage rather than a binary segmentation if variable stiffness modifiers are utilized. However, uncertainty quantification in such regression tasks is recommended because the robustness of the data-driven model depends on the quality of data and also the intensity of damage.

4.2. Vehicle Passage Dataset

For a more realistic simulation, it is essential to have reasonable and diverse values of the load imposed by a vehicle on the bridge. Furthermore, the distance between the axles (Wheelbase, WB) is also critical for properly modeling moving loads. The National Highway Traffic Safety Administration (NHTSA) is a valuable resource in this regard. We utilized the database of 528 passenger vehicles presented in the technical report from Bixel, Heydinger [40]. Wheelbase and gross axle weights of different models from 36 car manufacturers between the years 2000–2008 are considered as shown in Figure 5.

In the simulations, we utilized the Front and rear Gross Axle Weight Rating (GAWR), which represents the highest value an axle can support. The axle loads are scaled with a factor of 0.25 to be consistent with the quarter scale model in this numerical experiment. Considering the linear structural simulation of damaged and undamaged structure, appropriate scale analysis can be conducted for full-scale deployment of Trident. The distributions of the axle weight and wheelbase from the NHTSA database are given in Figure 6. The dataset is split into training, validation, and test sets based on alphabetic letters of car makes to minimize bias. The information about these splits is given in

Table 1. Passenger vehicle car makes in different data splits.

Split	Makes
Training	Acura, Audi, BMW, Buick, Cadillac, Chevrolet, Chrysler, Daewoo, Dodge, Ford, Honda, Hummer, Hyundai
Validation	Infiniti, Isuzu, Jaguar, Jeep, Kia, Land Rover, Lexus, Lincoln, Mazda
Test	Mercedes-Benz, Mini, Mitsubishi, Nissan, Pontiac, Saab, Saturn, Scion, Smart, Subaru, Suzuki, Toyota, Volkswagen, Volvo

.

The real loads can differ from the nominal values obtained from the NHTSA dataset.

Table 2. Ranges of sampling for different coefficients used for data diversification.

Parameter	Description	Range
${\sigma }_n$	Load intensity noise	[0.05, 0.02]
dtn	Sampled noise time steps (s)	[0.01, 0.02]
v	Vehicle speed (mph)	[10, 50]
α	Axle load modification factor	[0.8, 1.2]
nd	Number of damaged elements	[0, 10]

provides information on a series of diversification factors that are randomly sampled from a range to consider potential variations in loads and damage. The load intensity noise parameters, ${\sigma }_n$, and dt_n, are defined in Section 3. Vehicle speed (v) also determines the duration of loading (base of the triangular load pattern) for the deck nodes on the bridge. The axles do not always carry loads equal to their full capacity and the distribution of loads between the front and rear axles can vary compared with GAWR. The axle load is adjusted by a modifier coefficient (α) that is randomly sampled for each realization to consider this variation. Another parameter is the number of damaged elements (n_d) that is assumed to vary from 0 (no damage) to 10. The dataset is created by conducting 52,800 vehicle passing simulations, having 100 realizations for each vehicle model.

5. Performance Evaluations

This section contains Trident’s evaluation metrics for the bridge damage dataset mentioned earlier. The design of this model involves several hyperparameters that could substantially affect its robustness. The network starts with three ConvLSTM3D branches, followed by a concatenation of features and stacks of 3D convolution, batch normalization, and activations. Further details are provided in

Table 3. Trident layer specifications.

Layer	Hidden Units	Kernel Size	Padding
3 × ConvLSTM3D	125	(3,1,1)	-
Concatenation	-	-	-
3D Convolution	512	(3,3,3)	(1,1,1)
Batch normalization + ReLU	-	-	-
3D Convolution	512	(3,3,3)	(1,1,1)
Batch normalization + ReLU	-	-	-
3D Convolution	256	(3,3,3)	(1,1,1)
Batch normalization + ReLU	-	-	-
3D Convolution	256	(3,3,3)	(1,1,1)
Batch normalization + ReLU	-	-	-
3D Convolution	1	(3,3,3)	(1,1,1)
Sigmoid	-	-	-

.

The damage dataset is highly imbalanced towards the undamaged nodes, and proper selection of training hyperparameters is critical. We have utilized the focal loss function as described in what follows [41]:

$$L_f={\alpha }_t{(1-p_t)}^{\gamma }\log p_t$$

(7)

where:

$$p_t=\{\begin{array}{l}p\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}y=1\hspace{0.17em}(\mathrm{damage})\\ 1-p\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}y=0\end{array}$$

(8)

p values are Trident’s output probabilities for each structural node and ${\alpha }_t$ is a weight factor that depends on the target labels (y). The focal loss hyperparameters α and γ affect the model’s sensitivity to the imbalance.

$${\alpha }_t=\{\begin{array}{l}\alpha \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}y=1\hspace{0.17em}(\mathrm{damage})\\ 1-\alpha \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}y=0\end{array}$$

(9)

Learning rate, L2 regularization coefficient, scheduled learning rate decay coefficient, and decay step size are considered the other training hyperparameters. Bayesian optimization is utilized to tune these values and the ranges provided in

Table 4. Hyperparameter search space.

Hyperparameter	Search Intervals	Variable Type
α	(0.1, 1)	Real
γ	(0, 10)	Real
Learning rate	$({10}^{-5}$$, {10}^{-3}$)	Real
L2 regularization coefficient	$({10}^{-12}$$, {10}^{-4}$)	Real
Coefficient of learning rate decay	(0.2, 1)	Real
Step size for learning rate decay	(100, 2000)	Integer

.

A model may predict all nodes as damaged (D) or undamaged (ND) depending on the choice of hyperparameters. These circumstances are not desired as a model provides constant predictions regardless of input and is caused by improper training hyperparameters. Mean class accuracy, defined as the average of damage and undamaged classes, yields a 50% accuracy for such cases. As a more meaningful metric, we construct the confusion matrix for all nodes in the structure and take the geometric mean of D and ND classes for each node. The Node Average Geometric Mean Accuracy (NAGMA) is obtained by averaging the geometric means over all nodes in the structure. NAGMA obtained on the validation set is the objective function output of Bayesian optimization. Unlike mean class accuracy, NAGMA is 0 for models with constant predictions.

An early stopping patience of 3 epochs is considered to stop training before overfitting. The model with weights in the epoch with the lowest validation loss is selected for further testing, and a maximum of 50 epochs is considered for training. The performance metrics of the best model after 25 iterations are provided in

Table 5. Trident’s performance metrics for different data splits.

	Mean Node Accuracy		Global Accuracy
Data split	D	ND	D	ND	NAGMA
Training	84.59	98.17	81.34	98.27	90.99
Validation	77.04	96.56	72.61	96.76	86.02
Test	77.03	96.72	72.61	96.90	86.07

. The test confusion matrices for each structural node are also provided in Figure 7.

6. Conclusions

Structural health monitoring of bridges is critical for their safe operation as essential parts of the transportation infrastructure. The latest advances in technology and the availability of sensing equipment have resulted in the growing accessibility of continuous vibration monitoring of bridges. Despite such advances, processing massive amounts of vibration data to interpret damage, especially in near real-time, is valuable to stakeholders but remains a challenge due to the scarcity of resources. AI models have proven to be powerful tools in vibration-based SHM. Although, the existing models have limited capacity in effectively processing vibration records from large sensor arrays installed on 3D structures.

This paper proposes Trident, a deep learning architecture that enables high-resolution damage diagnosis of bridge structures by processing 6D vibration input tensors. The ConvLSTM3D cells in each branch of Trident automatically extract features from temporal and frequency variations of vibration signals while simultaneously considering the 3D geometry of the structure. Trident is validated on a case study conducted on a 3D steel truss bridge. A damage dataset is generated by conducting 52,800 vehicle passing simulations using the NHTSA vehicle database. Gaussian white noise with different sampling parameters is included in the dynamic load patterns for more realistic simulations. Furthermore, several variabilities in vehicle speed and gross axle weight rating are studied in data generation. Bayesian optimization is conducted to tune Trident’s hyperparameters and enables achieving 86% test accuracy for this complicated damage diagnosis problem.

Trident can serve for other SHM applications where large sensor arrays beyond accelerometers are utilized. Future work can be dedicated to the sensitivity of Tridents robustness to the amount of collected data. Pretraining Trident with techniques, such as self-supervised learning to reduce the need for massive, labeled data is also a promising area for future studies.