A Hybrid Learning Framework for Enhancing Bridge Damage Prediction

Abstract

Bridges are crucial structures for transportation networks, and their structural integrity is paramount. Deterioration and damage to bridges can lead to significant economic losses, traffic disruptions, and, in severe cases, loss of life. Traditional methods of bridge damage detection, often relying on visual inspections, can be challenging or impossible in critical areas such as roofing, corners, and heights. Therefore, there is a pressing need for automated and accurate techniques for bridge damage detection. This study aims to propose a novel method for bridge crack detection that leverages a hybrid supervised and unsupervised learning strategy. The proposed approach combines pixel-based feature method local binary pattern (LBP) with the mid-level feature bag of visual words (BoVW) for feature extraction, followed by the Apriori algorithm for dimensionality reduction and optimal feature selection. The selected features are then trained using the MobileNet model. The proposed model demonstrates exceptional performance, achieving accuracy rates ranging from 98.27% to 100%, with error rates between 1.73% and 0% across multiple bridge damage datasets. This study contributes a reliable hybrid learning framework for minimizing error rates in bridge damage detection, showcasing the potential of combining LBP–BoVW features with MobileNet for image-based classification tasks.

1. Introduction

Bridge infrastructure is critical to our transportation networks, and ensuring bridges’ structural integrity is primary. Deterioration and damage to bridges can lead to significant economic losses, traffic disruptions, and, in severe cases, loss of life because they are susceptible to various forms of damage, such as cracks, corrosion, and erosion, which can compromise their safety and functionality. Early detection of this damage is crucial for timely maintenance and repairs, preventing catastrophic failures, and minimizing economic losses [1,2].

In the past decade, several industries have employed artificial intelligence (AI) algorithms to acquire information and insights, thus enhancing their performance. The concept of AI is to provide machines with the ability to think and act like human beings. It contains supervised and unsupervised learning strategies. Artificial intelligence methodologies, including diagnosis, prediction, clustering, and classification, are used to develop models that distinguish between various data classes [3]. These models are employed to predict categorical class labels, such as “background” or “defects” and “crack” or “non-crack” in the classification of bridge damage. These techniques are employed to transform data into knowledge, enabling specialists to improve their work. This knowledge is represented as models. A model exhibiting high-performance metrics indicates that the artificial intelligence methodology effectively extracted knowledge and distinguished between various data patterns. However, the extracted knowledge may not always provide the correct predictions [4].

In the context of feature extraction based on a handcrafted method, Cao et al. [5] trained CNN descriptors using BoVW, while Hassan et al. [6] utilized color moments as a global feature and LBP as a local feature to construct a BoVW. Additionally, various studies [7,8,9,10] utilize BoVW for image classification and recognition. Many methods have been used to modify BoVW; for instance, an improved BoVW (iBoVW) has been proposed to generate the features from each convolutional layer of the pre-trained CNN model [11]. Also, a BoVW model presented by Olaode and Naghdy [12], whereby deep feature learning via a stacked autoencoder was used to extract image features and for video-based recognition, spatiotemporal scale-coded BoW (SC-BoW) was utilized. It entails splitting spatiotemporal characteristics into sub-groups according to the geographic scale from which they were taken and then encoding the acquired multi-scale information into BoW representations [13]. The SO-BoVW model proposed by Sultani and Dhannoon [14] combined the scale invariant feature transform (SIFT) and oriented FAST and rotated BRIEF (ORB) based on the BoVW model for image classification, where k-nearest neighbor (kNN) was utilized for image classification.

Based on our analysis of concrete damage detection models mentioned in previous literature, we have seen that some studies utilize deep learning models while others employ machine learning approaches. Some studies combine image processing techniques with either deep learning or machine learning. Bhalaji Kharthik et al. [15] employed twelve transfer learning DCNN models to categorize fractures in three publicly available datasets: CCIC, SDNET for structural cracks, and BCD for bridge fissures. The researchers applied two image enhancement techniques, LBP and contrast enhancement, to the SDNET images owing to the constrained efficacy of the transfer learning DCNN models. Furthermore, the support vector machine (SVM) was trained utilizing deep features derived from the final fully connected layer of the DCNNs. The integration of deep features with SVM resulted in an enhancement in detection accuracy across all combinations of DCNN and datasets. Yang et al. [16] proposed the utilization of the DCNN framework to classify fissures across three publicly accessible datasets: CCIC, BCD, and SDNET. The proposed methodology facilitates the transfer of three distinct forms of knowledge derived from prior scholarly achievements: sample knowledge, parameter knowledge, and model knowledge. The VGG network was augmented with additional fully connected layers, thereby establishing a novel learning framework. The researchers substantiated the credibility and efficacy of the proposed methodology. Zoubir et al. [17] employed a histogram of oriented gradients (HOG) in conjunction with uniform local binary patterns (ULBPs) to derive features from a dataset comprising over 3000 images, both uncracked and cracked, which encompassed various crack patterns and representations of concrete surfaces. Nonlinear dimensionality reduction techniques were executed through kernel principal component analysis (KPCA), and three machine learning classifiers were utilized to facilitate the classification process. The findings of the experiments indicate that the classification approach predicated on the SVM model and the feature-level fusion of HOG and ULBP features subsequent to the application of KPCA yielded optimal results, as the proposed classification framework accomplished an accuracy rate of 99.26%. Zoubir, with another team [18], introduced a publicly accessible benchmark-annotated image dataset comprising over 6900 images depicting cracked and non-cracked concrete bridges and culverts. The dataset presented encompasses a variety of challenging surface conditions and comprises concrete cracks exhibiting diverse sizes and patterns. The authors analyzed the proposed dataset employing three cutting-edge DCNNs, namely VGG16, VGG19, and InceptionV3, utilizing a transfer learning approach. The models above were employed to differentiate between cracked and non-cracked images, achieving a maximum testing accuracy of 95.89%. The outcomes of the experiments illustrate the promising applicability of this dataset for training deep learning networks aimed at the recognition of concrete cracks in bridge structures. Xu et al. [19] created a convolutional neural network (CNN)-based model for crack detection that works completely from the beginning to the end. It does this by using atrous convolution, the atrous spatial pyramid pooling (ASPP) module, and depthwise separable convolution. An implementation of atrous convolution makes it easier to obtain a wider receptive field while maintaining the input data’s resolution. When the ASPP module is used, the network can gather contextual data at different levels, and the depthwise separable convolution makes the computations easier. The model proposed by the authors achieved an impressive detection accuracy of 96.37% for the BCD dataset without the necessity of pre-training.

In this study, we aim to build a classification model that minimizes the prediction errors. At first, we built a model based on the LBP–BoVW, Apriori, and MobileNet algorithms. We achieved excellent accuracy with this model, ranging from 98.27% to 100%, and very few misclassified instances with an error rate between 1.73% and 0% for all datasets. The contributions of our work are summarized as follows:

• Reliable hybrid supervised and unsupervised learning based on LBP–BoVW features for minimizing error rate;
• A new algorithm for reliable learning on image datasets;
• Employing the Apriori algorithm for selecting robust features and dimensionality reduction.

2. Materials and Methods

The proposed model contains multiple steps: preprocessing, feature extraction based on LBP–BoVW, feature selection and dimensionality reduction based on the Apriori algorithm, and classification using MobileNetV3_Large algorithms. Figure 1 explains the general structure of the proposed model.

2.1. Preprocessing

Depending on the purpose, nature, quality, and use of the data, one can employ various strategies, as data preparation is a crucial step in preparing for model training. Preprocessing the model might also boost model inference and reduce training time. Reducing the size of the input images can significantly reduce the model’s training time while still preserving its performance, especially if the images are very large. This paper employs the following preprocessing techniques:

• Image resizing works the model correctly; images must be resized to a consistent size. A large image is difficult to process effectively. Therefore, we resized the different sizes of the used datasets to 100 × 100.
• Grayscale image: A grayscale instead of color is used in this work to simplify the image’s data and lower the processing requirements of the algorithms.
• Data augmentation techniques are used to produce new images from existing ones to increase the size of a dataset. It enhances the model’s generalization and decreases overfitting.
• The data normalization technique sets pixel intensity values to a predefined range, usually between 0 and 1.

2.2. Local Binary Pattern

The LBP is a textural descriptor described by Ojala et al. [20] and utilized for classification. To calculate a binary pattern, LBP compares the intensity of a center pixel (threshold) with the intensity of the 8 neighbor pixels around it. Figure 2 shows how the LBP works. LBP may be formalized as follows for a center pixel Ic and an adjacent pixel Ii (i = 1, 2, …, p) [21]:

$$LBP={\displaystyle \sum }_{i=1}^{p}F(I_i-I_c).2^{i-1}$$

(1)

$$F\left(I\right)=\left\{\begin{array}{c}1, \\ 0,\end{array}\right.\begin{array}{c}I\ge 0,\\ otherwise.\end{array}$$

(2)

2.3. LBP–BoVW

The LBP–BoVW features are extracted by training the image patches using the k-means clustering technique, where the size of the dictionary (k) is determined, which represents the number of clusters, and the cluster centroids are used as visual words. The steps of LBP–BoVW are explained as follows:

• Determine the number of patches needed to divide each image into patches. Here, we used 250 image patches.
• Compute the LBP for each patch.
• To determine the size of the dictionary (k), we used 50, 100, 200, and 300, which represent the number of K-means clusters.
• Identify each cluster’s center. These centers are the visual words. The size of the visual word vector is equal to K.
• Compute the histograms for each image to create the local feature vectors.

Figure 1 explains the steps above.

2.4. Feature Selection and Dimensionality Reduction

The hidden patterns and knowledge are extracted from massive amounts of data in the form of association rules, with the assistance of association rule mining. One of these association rule mining algorithms, the Apriori algorithm [22], uses frequent item set mining to generate association rules from a transactional dataset. Additionally, machine learning uses it for classification, feature selection, and dimensionality reduction. Selecting characteristic features from raw data is a fundamental stage in data mining. There are two main objectives for feature extraction. The first is to improve the classifier’s efficiency by representing each data object with a limited number of useful features, and the second is to remove features that are not representative of the data object, which will increase classification accuracy [23,24]. The main objectives of dimensionality reduction and feature selection are [25]:

• We can reduce overfitting by eliminating redundant or unnecessary features from the model, especially when working with high-dimensional data.
• Dimensionality and feature reduction can produce a simpler, more interpretable model that is easier to comprehend and explain.
• Reducing execution time by shrinking the model’s size, or data pruning, assists in speeding up the training and inference processes.
• By eliminating noisy or unnecessary features, the ability of a model to generalize to previously unobserved data can be enhanced.
• By data pruning, the best trade-off between model size and performance is achieved, resulting in a balance between accuracy and complexity.
• Feature reduction helps to enhance model performance and lower the chance of overfitting.

We used min-support = 0.9 and min-confidence = 0.9 as a threshold for feature selection. After applying the Apriori algorithm to the features derived from LBP–BoVW, the number of features that are selected in one image differs from the number of features that are selected in another image. So, the resulting features need to be processed to obtain an equal number of features for each image. To treat these missing values, we used the imputation mean method. When data are imputed, missing values are substituted with values that make sense given the information at hand. Data can be imputed in a way that maintains the set’s size and structure, prevents the removal of information that can be valuable, and lowers the predictability and bias of the estimations.

Table 1. Number of features after and before Apriori feature selection.

	Datasets	No. LBP–BoVW Features
		50	100	200	300
No. Features After Apriori Feature Selection	DIMEC-Crack Database	21	19	18	17
	BCD	21	17	14	14
	Bridge	20	19	18	18

explains the number of features after Apriori feature selection and the size of the LBP–BoVW features for the used datasets.

2.5. MobileNet

MobileNet is a computer vision model that has been made available as open source by Google. It is specifically developed for training classifiers. The network is a CNN that uses depthwise convolutions to greatly decrease the parameter count in comparison to other networks, thereby creating a lightweight deep neural network. MobileNets are compact models designed to accommodate limited resources, such as low latency and low power consumption. They are specifically parameterized to cater to a range of use cases, making them well suited for mobile applications and other situations with constrained resources. The efficiency of MobileNet is attained by using depthwise separable convolutions, which decrease the number of multiply-accumulates and the total size of the model. Although MobileNets exhibit superior speed and compactness compared to other prominent networks, they may sacrifice some accuracy in comparison to bigger, resource-intensive models. Nevertheless, its performance remains commendable, with just a little decline in accuracy [26]. In our work, we trained the datasets using MobileNetV3-Large [27].

3. Results and Discussion

3.1. Experimental Environment

In this section, we demonstrate that the proposed model is generalizable and capable of minimizing error rate by evaluating its performance on several datasets using 10-fold cross-validation.

Our model was implemented in the TensorFlow and Keras environments. We developed it in Python 3.9 using the Anaconda Navigator IDE and utilized it with Windows 10 64-bit. An i7-8550U CPU run at 1.99 GHz and 32 GB of RAM, and the experiment’s settings were as follows: the batch size was 128, the epoch was 20, and the optimization method for loss was Adam optimization.

3.2. Datasets

To test our model, we used four bridge damage datasets and compare our work with other studies that used these datasets. The experiment datasets will be described in this section.

• DIMEC-Crack Database: Lopez Droguett et al. [28] developed a new dataset for crack semantic segmentation. The dataset contains images extracted from video captured by an unmanned aerial vehicle (UAV) equipped with high-resolution cameras for several concrete bridges. Each image has a resolution of 1920 × 1080; in their study, they extracted non-overlapping patches of 96 × 96 pixels from each raw image, but we will use the original raw images. The dataset contains 10,092 high-resolution images separated into two classes: 7872 crack and 2220 non-crack images.
• Bridge concrete damage (BCD): Xu et al. [19] introduced a dataset that includes 2068 images of bridge cracks and non-cracks. They captured the images using the Phantom 4 Pro’s CMOS surface array camera, boasting a resolution of 1024 × 1024. The images underwent two reductions, first to dimensions of 512 × 512 and then to a size of 224 × 224 to produce the dataset. This dataset consists of 4057 photos with cracks and 2013 images without cracks.
• Bridge datasets: The dataset created by Zoubir et al. [17] yielded a total of 1304 cracked and 1806 non-cracked RGB bridge images at a resolution of 200 × 200. These images depict various concrete surfaces and cracks from the actual bridge examination. In order to minimize crack-like noise, the images were pre-processed using a 3 × 3 median filter after being converted to grayscale.

3.3. Evaluation Metrics

Because our models are binary classification, we utilized performance metrics that experiment with binary classification models, such as accuracy, precision, recall (also called sensitivity), f1-score, ROC–AUC score, and error rate. All these metrics are taken from the confusion matrix, which shows the classifier’s performance. Four elements comprise the confusion matrix: false positive (FP), false negative (FN), true positive (TP), and true negative (TN). Let us define:

• y = actual labels;
• $\widehat{y}$ = predicted labels;
• N = total number of samples;
• False positive (FP): Incorrectly predicted positives (actual negatives);

  $$FP= {\displaystyle \sum }_{i=1}^N(y_i=0 and {\widehat{y}}_i=1)$$

  (3)
• False negative (FN): Incorrectly predicted negatives (actual positives);

  $$FN= {\displaystyle \sum }_{i=1}^N(y_i=1 and {\widehat{y}}_i=0)$$

  (4)
• True Positive (TP): Correctly predicted positives;

  $$TP= {\displaystyle \sum }_{i=1}^N(y_i=1 and {\widehat{y}}_i=1)$$

  (5)
• True Negative (TN): Correctly predicted negatives.

  $$TN= {\displaystyle \sum }_{i=1}^N(y_i=0 and {\widehat{y}}_i=0)$$

  (6)
• The formula for evaluation metrics is as follows [29,30]:

  $$\mathrm{Accuracy}={\displaystyle \frac{\left(\mathrm{TP}+\mathrm{NT}\right)}{\left(\mathrm{TP}+\mathrm{NT}+\mathrm{FN}+\mathrm{FP}\right)}}$$

  (7)

  $$\mathrm{Precision}={\displaystyle \frac{\mathrm{TP}}{\left(\mathrm{TP}+\mathrm{FP}\right)}}$$

  (8)

  $$\mathrm{Recall}={\displaystyle \frac{\mathrm{TP}}{\left(\mathrm{TP}+\mathrm{FN}\right)}}$$

  (9)

  $$\mathrm{F}1-\mathrm{score}=2\times {\displaystyle \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$$

  (10)

  $$\mathrm{Error}\mathrm{Rate}={\displaystyle \frac{\left(\mathrm{FP}+\mathrm{FN}\right)}{\left(\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}\right)}}$$

  (11)

  $$ROC–AUC={\displaystyle \frac{SP+SE}{2}}$$

  (12)

  $$SP={\displaystyle \frac{TN}{TN+FP}}$$

  (13)

  where SP is the true negative rate (TNR), also called specificity, and SE is the true positive rate (TPR), also called sensitivity or recall [31].

3.4. Cross-Validation

For the experiment, we use 10-fold cross-validation, with 9 folds accounting for the training dataset, 1-fold for testing, and 10 process repetitions. Cross-validation is essential for addressing the issue of overfitting. The selection of cross-validation folds is essential. We aggregated the outcomes of all folds by means of averaging.

3.5. Discussion

In this section, we display a detailed analysis of the results of the proposed models on the utilized datasets. Many factors affect model performance, such as preprocessing operations and their vital role in classification performance, feature selection, dimensionality reduction, and selecting an appropriate classifier. Also, the number of features, image quality, and lighting conditions all impact classification performance. We evaluated the performance of the proposed algorithm using MobileNet model.

Table 2. The results of the proposed model.

Datasets	Performance Metrics	Number of Visual Words
		50	100	200	300
BCD	Accuracy	99.98	99.99	99.98	99.97
	Precision	99.97	100	100	100
	Recall	100	99.99	99.97	99.97
	F1-score	99.99	100	99.98	99.98
	ROC–AUC	99.97	99.99	99.98	99.98
	Error rate	0.02	0.01	0.02	0.03
Bridge dataset	Accuracy	99.82	99.84	99.96	99.97
	Precision	99.78	99.90	99.97	99.97
	Recall	99.81	99.75	99.94	99.97
	F1-score	99.79	99.83	99.95	99.97
	ROC–AUC	99.82	99.83	99.96	99.97
	Error rate	0.18	0.16	0.04	0.03
DIMEC-Crack	Accuracy	99.98	99.99	99.98	99.97
	Precision	100	100	100	99.99
	Recall	99.97	99.99	99.98	99.97
	F1-score	99.99	99.99	99.99	99.98
	ROC–AUC	99.99	99.99	99.99	99.97
	Error rate	0.02	0.01	0.02	0.03

explains the performance of the proposed model using difference evaluation metrics using four sizes of features for each dataset.

As shown in Table 2, the BCD dataset with feature sizes of 100 performed the best, with an accuracy of 99.99% and a high recall of 99.99%, with a precision and F1 score of 100%, ROC–AUC of 99.99% and a 0.01 error rate. The best performance was attained by the Bridge dataset with a feature size of 300, with 99.97% for all performance metrics and an error rate of 0.03. The DIMEC-Crack dataset with feature sizes of 100 performed the best, with a high precision reaching 100%, which means there were no false positives, and 99.99% for all performance metrics, with an error rate reaching 0.01%. Figure 3 shows the plot of the average of the best accuracy and loss. Figure 4 shows the confusion matrix of the best results for each dataset.

Table 3. The classification accuracy score comparison.

Refrences	Method	K-Fold CV	No. of Epochs	Datasets
				BCD	Bridge	DIMEC-Crack
Bhalaji Kharthik et al. [15]	VGG16	-	-	99.83	-	-
	VGG19			99.67	-	-
	Xception			99.67	-	-
	ResNet 50			99.67	-	-
	ResNet 101			99.5	-	-
	ResNet 152			99.83	-	-
	InceptionV3			99.83	-	-
	InceptionResNet V2			99.5	-	-
	MobileNet			99.83	-	-
	MobileNetV2			99.83	-	-
	DenseNet121			99.67	-	-
	EfficientNetB0			99.83	-	-
Yang et al. [16]	TL model	-	20	99.72	-	-
Zoubir et al. [18]	TL model	5-fold	10	-	95.89	-
Zoubir et al. [17]	HOG + ULBP + KPCA+ SVM	5-fold	-	-	99.29	-
Xu et al. [19]	Atrous convolution, ASPP, and depthwise separable convolution	-	300	96.37	-	-
MobileNetV3_Large		10-fold	20	82.14	72.59	91.51
Proposed method	LBP-BoVW + Appriori + MobileNetV3_Large	10-fold	20	99.99	99.97	100

presents the accuracy scores for each dataset and compares them with related studies that classify the same datasets. The difference in methodologies prevents a direct comparison with these studies, so we will only compare accuracy.

Bhalaji Kharthik et al. [15] employed 12 transfer learning (TL) methods: VGG16 achieved an accuracy of 99.83%, with more than 99% for each precision, recall, and F1-score; VGG19 achieved an accuracy of 99.67%, with a precision, recall, and F1-score of >99%; Xception achieved 99.67% and more than 99% for each precision, recall, and F1-score; ResNet 50 achieved an accuracy of 99.67%, with the precision, recall, and F1-score reaching >99%; ResNet 101 achieved an accuracy of 99.5%, a precision of 100%, a recall of 99%, and an F1-score of 99.5%; ResNet 152 gained an accuracy of 99.83% and >99% for each precision, recall, and F1-score; InceptionV3 achieved an accuracy of 99.83%, a precision of 99%, a recall of 99%, and an f1-score of 99%; InceptionResNet V2 achieved an accuracy of 99.5%, with a precision of 100%, a recall of 99%, and an F1 score reaching 99.5%; MobileNet achieved an accuracy of 99.83%, with more than 99% for each precision, recall, and F1-score; MobileNetV2 achieved an accuracy of 99.83%, with a precision, recall, and F1 score of >99%; DenseNet 121 achieved an accuracy of 99.67%, with a precision, recall, and F1-score of 100%; and the EfficientNetB0 model yielded an accuracy of 99.6% and a precision of 99.83%, with a precision, recall, and F1-score of >99%.

Yang et al. [16] also used transfer learning, utilizing 13 layers of VGG16 and 2 FC layers. They trained their proposal with BCD and CCIC datasets. For the BCD dataset, they achieved an accuracy of 99.72%, a precision of 96.46%, and an AUC of 99.99%. In their study, Zoubir et al. [17] used ULBP, HOG, KPCA, and SVM to classify concrete cracks in the self-made bridge dataset. They achieved an accuracy of 99.26%, a precision of 95%, a recall of 99.23%, and an F1 score of 99.12%. Zoubir, with another team [18], utilized the Bridge dataset, supplemented it with culvert images, and applied transfer learning to the CNN models. They achieved an accuracy of 94.89% in VGG16, 95.39% in VGG19, and 95.89% in InceptionV3. The InceptionV3 model produced the most favorable outcomes. The model proposed by Xu et al. [19] is based on the atrous spatial pyramid pooling (ASPP) module and depthwise separable convolution. They trained the BCD dataset on 300 epochs, achieving an accuracy of 96.37%, with a precision of 78.11%, 100% recall, and an F1-score of 87.71%. When we compared our proposal to previous studies and with the MobileNetV3_Large model, we found that it outperformed in all feature sizes.

4. Conclusions

This paper presents a method for enhancing bridge crack prediction based on LBP–-BoVW for feature extraction, an Apriori algorithm for selecting the strongest features and dimensionality reduction, and MobileNetV3_Large as a classifier. The proposed method achieved promising results for three bridge crack image datasets, with an accuracy rate of between 98.27% and 100% and an error rate of between 1.73% and 0%. Although we trained the data using 10-fold cross-validation, which tested all dataset samples with 20 epochs, our proposed system achieved promising results compared to previous studies that used the same datasets with more training epochs and did not use cross-validation or have a lower k-fold. We suggest using our proposed method on real-time datasets in future work.