Effective Hybrid Structure Health Monitoring through Parametric Study of GoogLeNet

Abstract

This paper presents an innovative approach that utilizes infused images from vibration signals and visual inspections to enhance the efficiency and accuracy of structure health monitoring through GoogLeNet. Scrutiny of the structure of GoogLeNet identified four key parameters, and thus, the optimization of GoogLeNet was completed through manipulation of the four key parameters. First, the impact of the number of inception modules on the performance of GoogLeNet revealed that employing eight inception layers achieves remarkable 100% accuracy while requiring less computational time compared to nine layers. Second, the choice of activation function was studied, with the Rectified Linear Unit (ReLU) emerging as the most effective option. Types of optimizers were then researched, which identified Stochastic Gradient Descent with Momentum (SGDM) as the most efficient optimizer. Finally, the influence of learning rate was compared, which found that a rate of 0.001 produces the best outcomes. By amalgamating these findings, a comprehensive optimized GoogLeNet model was found to identify damage cases effectively and accurately through infused images from vibrations and visual inspections.

1. Introduction

Structural health monitoring systems have always required the detection of structural deficiencies like corrosion, reduction in elasticity, or cracks. For a few decades, visual inspection by skilled professionals has been the primary method of identifying structural problems. This method is often time-consuming, risky, and expensive. As a result, researchers are becoming increasingly interested in computer-vision-based sensors for structural damage detection due to their great benefits over manual visual assessment. These sensors enable the use of appropriately calibrated thresholds in image processing operations like edge detection and crack identification.

The detection of structural defects, particularly cracks, is currently achieved using two basic image processing techniques: sequential image processing and image binarization. Images can provide a sufficient range of information [1]. Liu [2] suggested an image binarization technique that was accurate enough to find cracks. The sequential image processing technique was used to successfully identify cracks in photographs of concrete surfaces by Ebrahimkhanlou [3]. However, the majority of the edge detection techniques through various filters, including Roberts, Prewitt, Sobel, and Gaussian, have been used in vision-based fracture detection approaches for concrete structures [4]. These approaches are typically time-consuming and only capable of detecting the presence of cracks in a picture; they cannot assess the severity of cracks directly.

Recent research aimed to categorize and analyze structural defects like cracks and corrosion using new deep learning algorithms to overcome the shortcomings of vision-based crack detection methods. A CNN method was suggested by Rawat and Wang [5] for classifying images. With the use of a Nave Bayes data fusion approach, Chen and Jahanshahi [6] proposed a CNN-based fracture detection system and extracted cracks from video frames recorded for a structure at pixel-level precision. Dorafshan et al. [7] suggested a quicker CNN model with picture binarization. To identify and locate cracks, Soloviev et al. [8], Li et al. [9], Tong et al. [10], and Fan et al. [11] created deep CNN models to identify cracks and determine their sizes.

On the other hand, dynamic properties of a structure, such as its natural frequency, damping ratio, and mode shape, may also be utilized to evaluate the health of the structure and determine where damage has occurred. Structural degradation will alter the mass and stiffness distributions of structures, changing their dynamic properties [12,13,14]. Vibration-based structural damage detection methods with system identification could be utilized to establish correlations between the vibration features and the damage information [15,16,17], except natural frequencies are not sensitive to minor damage and could be easily contaminated by environmental factors. Following a similar thought process as the image-based structural condition assessment, some researchers also investigated identifying damage through deep learning models with vibration data inputs. Lin, Nie, and Ma [18] suggested a deep learning technique based on a CNN that has three maximum pooling layers and six convolutional layers. After that, the raw vibration data from a finite element beam were used to train this CNN-based system, and the damage was effectively identified with a high accuracy rate of 94.57%. Applying the CNN-based method, some researchers were able to find the damage created by losing bolts of a steel truss with great accuracy [19,20] and by using wireless sensor network inputs [21].

Artificial intelligence (AI) plays a transformative role in structural health monitoring by enabling the more accurate, efficient, and proactive management of infrastructure. Through advanced algorithms and machine learning techniques, AI can analyze vast amounts of data from sensors embedded in structures—such as bridges, buildings, and dams—to detect early signs of potential issues. This capability allows for the real-time assessment of structural integrity, identifying anomalies or patterns that might indicate wear, stress, or damage. Many researchers applied artificial intelligence and machine learning in the optimization and monitoring of construction activities [22,23,24]. Artificial intelligence models can assist construction safety monitoring and construction activity scheduling tremendously, which has helped to reduce construction accidents and shorten construction time [25].

In this research, our primary objective revolves around enhancing the performance of GoogLeNet when integrating fusion images derived from different vibration data and damage images. By merging these distinct data, it can create a robust and effective framework capable of detecting and diagnosing structural damage with heightened accuracy and efficiency. Through this fusion approach, we endeavor to exploit the complementary information present in both types of data, leveraging the spatial and spectral characteristics of damage images along with the dynamic signs captured by vibration data. At the same time, we aim to optimize GoogLeNet’s architecture to seamlessly integrate these heterogeneous inputs. Based on the above rationale, the manuscript is organized as follows: In Section 1, the literature review and background on structural health monitoring using fused data from different sources is summarized. In Section 2, the detailed GoogLeNet and its potential in SHM are included. In Section 3, a parametric study was conducted to identify the most efficient GoogLeNet, and their performance on fused data was compared. Section 4 provides the conclusion of the study.

2. Research Methodology

2.1. Deep Learning-Based Damage Detection Using the Fusion of Vibration Data and Defect Images

Typically, there are three main sections in a convolutional neural network to identify the structural damage through fusion data, as shown in Figure 1, including (1) data set collection, (2) CNN model training, and (3) validation and prediction. Vibrations at measure points under dynamic loads are first obtained in the ABAQUS model and converted to images. These vibration images are then paired with structure damage images to serve as inputs to the GoogLeNet model.

After that, the paired images are classified as different patterns according to different damage cases and are divided into three groups randomly, 70% of which are for training, 15% of which are for validation, and 15% of which are for testing.

The three layers that form the suggested CNN model are the convolution layer, pooling layer, and fully connected layer. Artificial neurons are stacked in these layers according to three dimensions: depth, width, and height. The damage identification part uses the paired images at various locations on the structure, which is represented as a two-dimensional matrix, as its input data. The CNN model then iteratively goes across each layer to convert the two-dimensional matrix into a one-dimensional vector matching to the category.

The fundamental component of the CNN model is the convolutional layer. Each convolutional block has variables that can be learned, such as biases and weights (filters). The filter’s depth is the same, but its width and height are spatially less than the input. For instance, since the RGB image contains three layers, the convolutional layer should have three depths to examine each of the three levels separately. The previous layer’s features are convolved using filters to create the output, which is then mapped to the new features in the current layer using the activation function. The convolutional layer for a pixel used the following equation:

$$C_{xy}=\sigma \left({\displaystyle \sum }_{i=1}^h{{\displaystyle \sum }}_{j=1}^w{{\displaystyle \sum }}_{k=1}^d{f}_{ijk}\ast X_{\mathit{x}+i,y+j-1,k}+b\right)$$

(1)

where X is the input vector of intensity values of the image, b is the bias vector, σ is the activation function, and f is the weight matrix, and where h and w are the filter’s sizes and d is the number of input channels.

Figure 2 illustrates the convolution process on a randomly chosen 4 × 4 matrix. A 3 × 3 matrix is selected as the filter which is generated randomly at the initial state and updated from the model by a backward propagation algorithm.

Four sub-arrays of the same size are created by sliding along the input matrix’s width and height with a stride of 1. Each sub-array is multiplied by the filter matrix. The output value, which is the result of adding the multiplied values and the bias, is then obtained. Because of the stride, the output is smaller than it was in the prior layer. The nonlinear activation function, which is used to add nonlinearity to the network, comes after the convolution layer. Rectified linear unit (ReLU), softplus, tanh, and Elu are some of the activation functions that are most frequently employed in neural networks. Their respective functions are expressed as follows:

$$f\left(x\right)=\max \left(0,x\right)$$

(2)

$$f{\left(x\right)}_i=log(1+e^x)$$

(3)

$$f\left(x\right)={\displaystyle \frac{e^x-e^{-x}}{e^x+e^{-x}}}$$

(4)

$$f\left(x\right)=\max \left(e^x-1,x\right)$$

(5)

where f(x) is the function that could represent the probability of firing or activating the neurons in the next inception layer and x denotes the state of the current neuron. These functions are shown in Figure 3.

The optimizer is considered one of the crucial components of the training process. The optimizer is responsible for adjusting the parameters of the network during the training process to minimize the loss function. Many optimizers can be used in training, such as stochastic gradient descent (Sgdm), adaptive momentum estimation (Adam), and root mean square propagation (RmsProp). The choice of optimizer can significantly impact the convergence speed and performance of the model in the task at hand. Therefore, selecting an appropriate optimizer and tuning its parameters are essential steps in the development of a successful CNN model.

The final layer before the network’s output layer is the completely connected layer. All of the neurons in this layer are connected to the features created in the layer before. The generated features are transformed into corresponding categories in this layer via weights and biases. The equation of the output yⁱ is shown as follows:

$$y^l=\sigma \left(y^{l-1}\ast w+b\right)$$

(6)

where σ is the activation function; w and b represent the weights and bias vectors in this layer, respectively; i is the iteration step.

2.2. Transfer Learning Method Used

GoogLeNet is a particular kind of convolutional neural network built on the inception model. Using inception modules, the network is able to select among a variety of convolutional filter sizes for each block. These modules are stacked on top of one another using a convolutional network, which occasionally uses max-pooling layers with stride 2 to cut the grid’s resolution in half.

GoogLeNet has been trained on over a million images and can classify images into 1000 object categories (such as keyboards, coffee mugs, pencils, and many animals). The network has learned rich feature representations for a wide range of images. The network takes an image as input with 224 × 244, and then outputs a label for the object in the image together with the probabilities for each of the object categories.

The GoogLeNet architecture consists of 22 layers (27 layers including pooling layers) which are tabulated in

Table 1. The architecture of the GoogLeNet algorithm.

Type	Patch Size Stride	Output Size	Depth
Convolution	7 × 7/2	112 × 112 × 64	1
max pool	3 × 3/2	56 × 56 × 64	0
Convolution	3 × 3/1	56 × 56 × 192	2
max pool	3 × 3/2	28 × 28 × 192	0
inception (3a)	-	28 × 28 × 256	2
inception (3b)	-	28 × 28 × 480	2
max pool	3 × 3/2	14 × 14 × 480	0
inception (4a)	-	14 × 14 × 512	2
inception (4b)	-	14 × 14 × 512	2
inception (4c)	-	14 × 14 × 512	2
inception (4d)	-	14 × 14 × 528	2
inception (4e)	-	14 × 14 × 832	2
max pool	3 × 3/2	7 × 7 × 832	0
inception (5a)	-	7 × 7 × 832	2
inception (5b)	-	7 × 7 × 1024	2
avg pool	7 × 7/1	1 × 1 × 1024	0
Dropout (40%)	-	1 × 1 × 1024	0
Linear	-	1 × 1 × 1000	1
soft max	-	1 × 1 × 1000	0

, and part of these layers are a total of nine inception modules, as shown Figure 4. Each red circle represents an inception layer.

2.3. Gramian Angular Field (GAF)

The GAF algorithm was developed in the area of time series data to preserve characteristics when encoding 1D time series into 2D images. The two methods that can be used to complete GAF are the Gramian angular difference field (GADF) and the Gramian angular sum field (GASF). The vibration data are X = {x₁, x₂, …, x_i, …, x_n}, where n is the total number of displacement points and x_i is the value of the displacement. The spectral data must first be normalized, scaled to the range of [0, 1], and recorded as x_i,n.

$${x_i}_n={\displaystyle \frac{x_i-min\left(X\right)}{max\left(X\right)-min\left(X\right)}}$$

(7)

Next, using the following formula, the 1D spectral sequence in the Cartesian coordinate system is converted to a polar coordinate system:

$$\phi =arccos\left(x_{i,n}\right)$$

(8)

Time dependence is maintained in the Gramian matrix. The time dimension is encoded into the matrix’s geometry because time grows as the position shifts from the upper left corner to the lower right corner. Equation (8) shows that the converted angle ϕ has a value range of [0, π] and that the cosine value decreases monotonically within this range. The increase in time corresponds to the radius in the polar coordinate system, while x_i in each Cartesian coordinate system responds to the angle value in the polar coordinate system, and the corresponding bending takes place between various angle points on the polar coordinate circle. Thus, the Gramian angular summation field (GASF) can be obtained by computing the cosine value of the sum of angles between various points, as follows (Equation (9)):

$$GASF=\left[\cos \left({\phi }_i+{\phi }_j\right)\right]=\left[\begin{array}{ccc}\cos \left({\phi }_1+{\phi }_1\right)& \cdots & \cos \left({\phi }_1+{\phi }_n\right)\\ ⋮& \ddots & ⋮\\ \cos \left({\phi }_n+{\phi }_1\right)& \cdots & \cos \left({\phi }_n+{\phi }_n\right)\end{array}\right]={\tilde{x}}_i.{\tilde{x}}_j-\sqrt{1-{\tilde{x}}_i^2}\sqrt{1-{\tilde{x}}_j^2}={\tilde{X}}^T\tilde{X}-{\sqrt{1-{\tilde{X}}^2}}^T\sqrt{1-{\tilde{X}}^2}$$

(9)

where I is the unit vector. ${\tilde{X}}_i$ is the normalized vibration data, and φ is the angle of the point.

An example employing vibration data from the steel truss model’s ABAQUS model is shown in Figure 5a. The suggested GASF is used to transform the vibration into images, and the resulting images include several distinct elements, including the lines and points that are indicated in Figure 5b. Red color indicates higher value locations while blue colors stand for the low value locations and green colors stand for the intermediate value locations.

3. Case Study and Validation

3.1. Data Collection through ABAQUS Modeling

The ABAQUS truss model is shown in Figure 6. In this study, the truss comprises eight spans and nine measurement points. There are 16 scenarios in all, and

Table 2. Damage positions for different cases.

case

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

span

-

S1

S2

S3

S4

S1

S2

S1

S2

S3

S4

S1

S2

S3

S4

S5

face

-

f

f

f

f

t

t

f

f

f

f

f

f

f

f

f

element

-

T

T

T

T

m

m

m

m

m

m

r

r

r

r

m

displays the locations of the damage in each case. f represents the front face and t represents the top face of the truss. T, m, and r represent the top element, middle element, and right element, respectively. For example, for case number 8, the sloping rod on the front face is damaged by reducing 30% of its elastic modulus. Case no. 5 represents the damage on the top element of the front face at the fourth span of the truss.

Dynamic load is applied at the mid-point of the truss, and dynamic displacement data are collected at the specified measurement point. Figure 7 shows the dynamic displacement at the MP1 location for damage case 2.

3.2. Structural Evaluation Using a Novel Deep Learning-Based Method with Both Defect Image and Vibration Data

As seen in Figure 8, pictures of structural defects at measurement points can be obtained using a camera or UAV. An image-based displacement sensor, on the other hand, gathers the structure’s vibration data. These one-dimensional vibration data are further converted into two-dimensional images using the above GASF algorithm. These two groups of images are then paired to provide input data for the deep learning model.

3.3. Gramian Angular Field

Without losing any features, 1D time series were encoded into 2D images using the GASF algorithm. An example using vibration data from the steel truss model in ABAQUS is presented in Figure 9.

3.4. Paired Images

Images of the defected truss for each case were taken, and vibration signals at each sensor location were recorded and converted to images using GASF. In addition, 288 paired images with sizes of 224 × 224 were obtained for the truss. Figure 10 shows a paired image of the truss and the image obtained by GASF.

4. Results of the Proposed Method

In order to validate this procedure, vibrations from the steel truss ABAQUS models were translated into images using GASF. In the ABAQUS model of the steel truss, there were sixteen sets of integrated images with various faults (30% reduction in modulus of elasticity at different locations) and damage locations (no damage, No. 3 damage, and No. 5 damage). There were 128 images in total and 9 images for each group of the case study.

4.1. Results of Model Training, Validation, and Prediction

The above-described process resulted in satisfactory outputs. For the damage truss cases at different locations with the combination of GASF images converted from vibration data, the accuracy and the loss of the training and validation of the GoogLeNet model are shown in Figure 11, which gives 100% accuracy. Figure 12 is shown the confusion matrix of using infused images as input data for the model. The solid line indicates the training accuracy while the dash line indicates the validation accuracy.

4.2. Effect of the Number of Inception Layers

Inception modules are important in feature detection. The training accuracy of GoogLeNet with different numbers of inception modules is shown in Figure 13. The results demonstrated the efficiency of the original configuration with nine inception layers in capturing intricate characteristics and patterns of the input data, with perfect 100% accuracy. Similarly, when reducing the number of inception layers to eight, the model maintained its high accuracy of 100%, indicating the robust performance of the GoogLeNet model even with a slightly simplified architecture. On the other hand, as the number of layers decreased further, the accuracy of the GoogLeNet model fluctuated. With seven layers, the accuracy dropped to 70%, suggesting a sharp decrease in performance due to its reduced capacity to capture intricate details. A further reduction to six layers led to a decrease in accuracy to 57%, highlighting the diminishing capacity of the model to learn and generalize the input data patterns. Overall, these results demonstrate the intricate relationship between model depth and performance, emphasizing the importance of striking the right balance between complexity and generalization capacity.

4.3. Effect of Learning Rate

Model accuracy is affected by the variation in learning rates, as shown in Figure 14. When utilizing a learning rate of 0.001, the model achieved 100% accuracy, showing the effectiveness of a smaller learning rate and steady convergence. However, when the learning rate was increased to 0.01, the accuracy dropped to 73%. While still respectable, this decrease suggests that the higher learning rate might have caused oscillations around the optimal solution or even led to divergence in some cases, hindering the model’s ability to converge effectively. Subsequently, employing a learning rate of 0.03 or more resulted in accuracy of a mere 7%, indicative of overshooting and divergence, as the excessively large learning rate likely caused the optimization process to become saturated and unstable, preventing the model from learning meaningful representations. These results underscore the critical role of learning rate selection in optimizing model performance, emphasizing the need for careful tuning to strike a balance between convergence speed and stability.

4.4. Effect of Activation Function

Figure 15 shows the effect of different activation functions, which gave different accuracies in model training. When employing Rectified Linear Unit (ReLU), the model achieved perfect 100% accuracy, indicating its effectiveness in promoting nonlinearity and mitigating the vanishing gradient problem, thus enabling robust learning. Similarly, when using Exponential Linear Unit (ELU), the model again attained 100% accuracy. ELU’s capability to handle negative values might have facilitated smoother gradients and enhanced model convergence. Softplus activation function resulted in 25% accuracy, likely due to its gradual gradient and lack of a strict zero threshold, which might have hindered the model’s ability to capture complex patterns effectively. However, when employing Hyperbolic Tangent (tanh), the accuracy dropped to 7%. Tanh’s saturating nature might have led to vanishing gradients, impeding the model’s learning process and hindering its performance. In general, the divergent outcomes underscore the significant impact of activation functions on model learning dynamics and performance.

4.5. Effect of Different Optimizers

Training GoogLeNet with different optimizers yielded varying accuracies, as shown in Figure 16. When trained with Stochastic Gradient Descent with Momentum (SGDM), the model achieved remarkable 100% accuracy, showing its ability to converge quickly and effectively optimize the parameters. However, when trained with Adam optimizer, the accuracy dropped to 67%. Adam optimizer, while popular for its adaptability to various datasets, might have struggled to navigate the complex architecture of GoogLeNet effectively, leading to suboptimal results. On the other hand, training with RMSprop yielded accuracy of a mere 7%, indicating its limitations in handling the intricate nuances of GoogLeNet’s architecture. RMSprop, though robust in handling non-stationary objectives, might have encountered difficulties in maintaining the necessary momentum for convergence in this scenario. Overall, the varying performances highlight the sensitivity of deep learning models to the optimizer selection, emphasizing the importance of careful experimentation and tuning to achieve optimal results.

4.6. Effect of Training Data Set Sizes

The effect of training data set sizes was studied and is shown in Figure 17. Here, 50%, 60%, 70%, 80%, and 90% of the total data were used as the training set, while the rest of the data were split equally into the validation and test sets. The accuracies at different iteration steps are also included in the figure. From the results shown, one can see the GoogLeNet model converges earlier when a larger portion of the data is used as the training data set. And the accuracy at the 80th iteration converges for all training data sizes and the accuracy at the 80th iteration can be used as a reliable criterion to analyze the effect of different hyperparameters.

5. Conclusions

In conclusion, this paper aimed to enhance the efficiency and accuracy of a new structure health monitoring method that utilizes infused images from vibrations and visual inspections through a parametric study of the GoogLeNet architecture. By leveraging deep learning techniques, this study sought to optimize the performance of the network by manipulating four key parameters: the number of inception modules, activation function, optimizer, and learning rate. Notably, employing eight inception modules achieved superior performance with less computational time compared to using nine inception layers. Additionally, the Rectified Linear Unit (ReLU) activation function emerged as the most effective choice, while the Stochastic Gradient Descent with Momentum (SGDM) optimizer and a learning rate of 0.001 showed superior performance. By integrating these findings, an efficient and effective new structural health monitoring method was found through optimization of the GoogLeNet model. This study not only provides valuable insights into structural health monitoring techniques but also underscores the importance of deep learning models to achieve superior model performance.

In future work, case studies with real structures of different damage conditions will be conducted. The effects of paired image sizes, other hyperparameters of the GoogLeNet model, and different sizes of training data set are not studied in this manuscript, and we hope to explore them.