Concrete Bridge Crack Image Classification Using Histograms of Oriented Gradients, Uniform Local Binary Patterns, and Kernel Principal Component Analysis

Round 1

Reviewer 1 Report

In this article, the authors use a dimensionality reduction technique to generate a feature subspace in order to solve the problem of time-consuming training and high technical cost. Linear and non-linear dimensionality reduction are performed using Principal Component Analysis (PCA) and its nonlinear kernel version (KPCA). Finally, (Support Vector Machine)SVM is used for classification.

Advantages:

1. The authors experimented to do sufficient experiments.

2. The innovative points of the article are also clear.

Disadvantages:

1. The authors make no reference in the text to relevant studies done by others. It would have been better for the reader to understand the current state of research in the field and to reflect the enhancements made by the authors.

Author Response

REVIEWER #1

Dear reviewer, Thanks a lot for your appreciated comment. The suggestion provided by the reviewer is really helpful in improving the overall quality of the manuscript.

Comment 1: The authors make no reference in the text to relevant studies done by others. It would have been better for the reader to understand the current state of research in the field and to reflect the enhancements made by the authors.

Authors response: We very much appreciate the Reviewer’s comment and we have made the necessary changes in the manuscript.

Authors actions: Dear reviewer, we are very grateful for your suggestion. We have updated the introduction section and discussed previous works related to crack detection in the context of handcrafted features-based frameworks. We have also clearly listed the main contributions of our work.

In the context of handcrafted features-based frameworks, Choudhary and Dey [13] used a Sobel edge detector to extract features from digital images of concrete surfaces and trained Neural Network and Fuzzy models to detect concrete cracks.

Chen et al. proposed a detection method for highway pavement damage [14]. The authors used a grayscale-weighted Histogram of Oriented Gradient feature patterns and a Convolutional Neural Network classifier.

 Jin et al. [15] used the Histogram of Oriented Gradients and the watershed algorithm to detect different pavement cracks and identify their severity. 

Another example was proposed by Zalama et al. [16]. The authors developed a methodology utilizing Gabor filters, several binary classifiers, and an Adaboost algorithm to detect longitudinal and transverse pavement cracks.

Chen et al. [17] presented a texture-based video processing approach using Local Binary Patterns (LBP), Support Vector Machine (SVM), and Bayesian decision theory to automate crack detection on metallic surfaces. Finally, Hu et al. [18] proposed a pavement crack detection method based on texture analysis using the Gray-Level Co-Occurrence Matrix, translation invariant shape descriptors, and an SVM classifier. 

Depending on the dimension of the extracted features, classifier training can be time-consuming and computationally expensive. This limitation can be addressed by applying dimensionality reduction techniques. For example, Principal Component Analysis, Linear Discriminant Analysis, and Isometric mapping can be used as a processing step in the general classification framework. In this sense, Abdelmawla et al.[19] applied image processing techniques, Principal Component Analysis (PCA), and K-means algorithm to detect and classify different types of cracks in pavement surface images.

In addition, Abdel-Qader et al. [20] proposed an algorithm based on PCA, linear features, and local information to detect cracks in concrete bridge images. Kumar et al. [21] used PCA and a modified LeNet5 model to detect cracks in roads and bridges working with three different crack datasets. Endri et al. [22] used PCA as a preprocessing step to extract features from a pavement crack dataset containing 400 images. Chen et al. [23] extracted features from frames of road videos using LBP, reduced the dimension of the LBP feature space by PCA, and trained an SVM classifier to detect different types of pavement cracks. Elhariri et al. [24] proposed a crack detection model for historical buildings images based on feature extraction, a fusion of handcrafted features (i.e., HOG and LBP features) and Convolutional Neural Network-learned features, dimensionality reduction using PCA and Machine Learning classifiers. 

In this paper, several concrete bridge crack classification schemes based on the Histogram of Oriented Gradients (HOG), Uniform Local Binary Patterns (ULBP), Kernel Principal Component Analysis (KPCA), and Machine Learning classifiers (i.e., SVM, Random Forests and Decision Trees) are studied and compared. The main contributions of this work are the following: 

• More than 3000 images of a crack dataset constructed by the authors in [25] are preprocessed using a Median Filter to remove crack-like noise; 
• HOG and ULBP features are extracted from the preprocessed images, and dimensionality reduction by KPCA is applied to the extracted features; 
• Classification schemes based on the reduced HOG and ULBP features and Machine Learning classifiers are investigated and evaluated using different classification metrics to yield the best model for bridge concrete crack detection.  

Author Response File: Author Response.pdf

Reviewer 2 Report

(1) The machine learning (ML) method used in this paper is a bit too old. At present, there are many algorithms with better performance that can be applied to this problem. This paper has certain practical significance, but the results are by no means the best. The author can consider the t-SNE manifold learning method in literature doi.org/10.1155/2022/2713386.

(2) There is a writing error "PCA of KPCA" in Figure 5(b), the same error is found in line 198, and there are many writing errors in the text.

(3) The indicators of formulas (4) - (7) are not applicable to the multi classification problem, but to the two classification problem. Please re-evaluate using the multi classification indicators Macro-F1 and Micro-F1.

(4) The serious problem in this paper is that SVM is a two classifier. How does it perform multi classification? Generally speaking, SVM needs to cooperate with multi classification strategy OVO, OVR or MVM for multi classification processing.

(5) There is another doubt. Please provide the with more data support. PCA processing nonlinear structural data will cause serious feature crowding. It is impossible to have such a small gap between the experimental results and KPCA results.

Author Response

REVIEWER #2:

Dear reviewer, Thanks a lot for you encouraging comments. The suggestions provided by the reviewer are really helpful in improving the overall quality of the manuscript.

Comment 1- The machine learning (ML) method used in this paper is a bit too old. At present, there are many algorithms with better performance that can be applied to this problem. This paper has certain practical significance, but the results are by no means the best. The author can consider the t-SNE manifold learning method in literature doi.org/10.1155/2022/2713386.

Authors response: We thank the reviewer for this comment. We have made the necessary changes to improve the results of our work.

Authors actions: We very much appreciate your comment.  We have conducted further experiments to improve the results of our work. In the updated version of our paper, we preprocessed the images using a Median Filter to remove crack-like noise. We also used two feature descriptors (Histogram of Oriented Gradients and Uniform Local Binary Patterns) to extract features from filtered images. We finally applied Kernel Principal Component Analysis for dimensionality reduction and fused the reduced features that we fed to Machine Learning Classifiers. The best classification scheme achieved an accuracy of 99.26%.

Please find below the updated abstract of our paper.

New Abstract:

Bridges deteriorate over time, which requires continuous monitoring of their condition. There are many digital technologies for inspecting and monitoring bridges in real-time. In this context, computer vision extensively studies cracks to automate their identification in concrete surfaces, overcoming the conventional manual methods relying on human judgment. The general framework of vision-based techniques consists of feature extraction using different filters and descriptors and classifier training to perform the classification task. However, training can be time-consuming and computationally expensive, depending on the dimension of the features. To address this limitation, dimensionality reduction techniques are applied to extracted features, and a new feature subspace is generated. This work uses a Histogram of Oriented Gradients (HOG) and Uniform Local Binary Patterns (ULBP) to extract features from a dataset containing over 3000 uncracked and cracked images covering different patterns of cracks and concrete surface representations. Non-linear dimensionality reduction is performed using Kernel Principal Component Analysis (KPCA), and three Machine Learning classifiers are implemented to conduct classification. The experimental results show that the classification scheme based on the Support Vector Machine (SVM) model and feature-level fusion of HOG and ULBP features after the KPCA application provided the best results, as an accuracy of 99.26% was achieved by the proposed classification framework. 

Comment 2- There is a writing error "PCA of KPCA" in Figure 5(b), the same error is found in line 198, and there are many writing errors in the text.

Authors response: We thank the reviewer for this comment. We have made the necessary changes in the manuscript.

Authors actions: Thank you for your comment. A complete proofreading has been made for the entire manuscript.

Comment 3 : The indicators of formulas (4) - (7) are not applicable to the multi classification problem, but to the two classification problem. Please re-evaluate using the multi classification indicators Macro-F1 and Micro-F1.

Authors response and actions : We thank the reviewer for this comment. We used the indicators of the mentioned formulas as we are working on a binary classification problem (Cracked and uncracked images).

Comment 4 : The serious problem in this paper is that SVM is a two classifier. How does it perform multi classification? Generally speaking, SVM needs to cooperate with multi classification strategy OVO, OVR or MVM for multi classification processing.

Authors response and actions: We thank the reviewer for this comment. We used the SVM classifier as we are working on a binary classification problem (Cracked and uncracked images).

Comment 5: There is another doubt. Please provide with more data support. PCA processing nonlinear structural data will cause serious feature crowding. It is impossible to have such a small gap between the experimental results and KPCA results.

 Authors response: We very much appreciate the reviewer’s comment. We have made the necessary changes in the manuscript.

Authors actions :  Thank you for your comment. We have reconducted the experiments several times and the gap between PCA and KPCA results remains the same.  In the updated version of the manuscript, we only used the Kernel Principal Component Analysis as the dimensionality technique applied to HOG and ULBP feature sets.

Author Response File: Author Response.pdf

Reviewer 3 Report

Dear authors, 

The work presented in this study considers only the traditional features (HOG) to classify the considered images into healthy/crack  using a binary classifier. The results obtained is considerably low (Accuracy =84.03%).

Please clearly justify, how this result can be increased >90 or 95% in the future. Please list all the features that can be accounted for along with HOG to achieve better detection.

Please consider the following suggestions:

1. In the introduction section, please discuss the major contributions of the presented study.

2. Include a detailed block diagram section in the Experimental setup which includes; image collection, image pre-processing, feature extraction, reduction, classifier training and validation, etc.

Please discuss the number of HOG features extracted. Also, discuss how the problem of overfitting is solved. (Feature reduction is necessary). Please include the number of original features and reduced features. Also include the k-fold cross-validation employed in this work. 

3. The results presented in Table 3 is not sufficient. Only SVM alone will not provide the best result in a chosen classification task. Please repeat this work with other classifiers (Ex. Random Foreset, Decision tree, K-nearest neighbors, etc) and present a comparative result.

4. The references are too small. In the literature, a number of building crack/damage detection works are available and earlier works can be included in references.

5. Overall comment: The results presented in this work needs a major improvement and I request you to improve it considerably.

Author Response

REVIEWER #3:

Dear reviewer, Thanks a lot for your encouraging comments. The suggestions provided by the reviewer are really helpful in improving the overall quality of the manuscript.

Comment 1- The work presented in this study considers only the traditional features (HOG) to classify the considered images into healthy/crack using a binary classifier. The results obtained is considerably low (Accuracy =84.03%). Please clearly justify, how this result can be increased >90 or 95% in the future. Please list all the features that can be accounted for along with HOG to achieve better detection.

Authors response: We thank the reviewer for this comment. We have made the necessary changes to improve the results of our work.

Authors actions: Thank you for your comment.  We have conducted further experiments to improve the results of our work. In the updated of our paper, we preprocessed the images using a Median Filter to remove crack-like noise. We also used two feature descriptors (Histogram of Oriented Gradients and Uniform Local Binary Patterns) to extract features from filtered images. We finally applied Kernel Principal Component Analysis for dimensionality reduction and fused the reduced features that we fed to Machine Learning Classifiers. The best classification scheme achieved an accuracy of 99.26%.

Please find below the updated abstract of our paper.

New Abstract:

Bridges deteriorate over time, which requires continuous monitoring of their condition. There are many digital technologies for inspecting and monitoring bridges in real-time. In this context, computer vision extensively studies cracks to automate their identification in concrete surfaces, overcoming the conventional manual methods relying on human judgment. The general framework of vision-based techniques consists of feature extraction using different filters and descriptors and classifier training to perform the classification task. However, training can be time-consuming and computationally expensive, depending on the dimension of the features. To address this limitation, dimensionality reduction techniques are applied to extracted features, and a new feature subspace is generated. This work uses a Histogram of Oriented Gradients (HOG) and Uniform Local Binary Patterns (ULBP) to extract features from a dataset containing over 3000 uncracked and cracked images covering different patterns of cracks and concrete surface representations. Non-linear dimensionality reduction is performed using Kernel Principal Component Analysis (KPCA), and three Machine Learning classifiers are implemented to conduct classification. The experimental results show that the classification scheme based on the Support Vector Machine (SVM) model and feature-level fusion of HOG and ULBP features after the KPCA application provided the best results, as an accuracy of 99.26% was achieved by the proposed classification framework. 

Comment 2- In the introduction section, please discuss the major contributions of the presented study.

Authors response: We thank the reviewer for this comment. We have made the necessary changes in the manuscript.

Authors actions: Thank you for your comment. We have updated the introduction section and we have listed the main contributions of our work. 

In this paper, several concrete bridge crack classification schemes based on the Histogram of Oriented Gradients (HOG), Uniform Local Binary Patterns (ULBP), Kernel Principal Component Analysis (KPCA), and Machine Learning classifiers (i.e., SVM, Random Forests and Decision Trees) are studied and compared. The main contributions of this work are the following: 

• More than 3000 images of a crack dataset constructed by the authors in [25] are preprocessed using a Median Filter to remove crack-like noise; 
• HOG and ULBP features are extracted from the preprocessed images, and dimensionality reduction by KPCA is applied to the extracted features; 
• Classification schemes based on the reduced HOG and ULBP features and Machine Learning classifiers are investigated and evaluated using different classification metrics to yield the best model for bridge concrete crack detection.  

Comment 3: Include a detailed block diagram section in the Experimental setup which includes; image collection, image pre-processing, feature extraction, reduction, classifier training and validation, etc. Please discuss the number of HOG features extracted. Also, discuss how the problem of overfitting is solved. (Feature reduction is necessary). Please include the number of original features and reduced features. Also include the k-fold cross validation employed in this work.

Authors response : We thank the reviewer for this comment. We have made the necessary changes in the manuscript.

Authors actions: Thank you for your suggestion. We have added a block diagram in the experimental setup section that describes the steps followed to implement our proposed classification method:

We have also presented the numbers of original HOG and ULBP features and reduced features in table 2 :

Table 2. Reconstruction error of HOG and ULBP features using KPCA

We have also mentioned that the performance of the classification methods is evaluated using a 5-fold cross validation procedure.

The performance of the classification methods is evaluated using a 5-fold cross validation procedure based on the following metrics:

                                                                                           (5)

                                                                                                        (6)

                                                                                                              (7)

                                                    (8)

Comment 4 : The results presented in Table 3 is not sufficient. Only SVM alone will not provide the best result in a chosen classification task. Please repeat this work with other classifiers (Ex. Random Forest, Decision tree, K-nearest neighbors, etc.) and present a comparative result.

Authors response: We very much appreciate the reviewer’s comment. We have made the necessary changes in the manuscript.

Authors actions: Thank you for your suggestion. We have conducted further experiments using Random Forests and Decision Trees classifiers and compared their results with the SVM model.

Kindly refer to the updated “Results and discussions” section.

Comment 5: The references are too small. In the literature, a number of building crack/damage detection works are available and earlier works can be included in references.

 Authors response: We very much appreciate the reviewer’s comment. We have made the necessary changes in the manuscript.

Authors actions :  Dear reviewer, we are very grateful for your suggestion. We have updated the introduction section and discussed previous works related to crack detection in the context of handcrafted features-based frameworks.

In the context of handcrafted features-based frameworks, Choudhary and Dey [13] used a Sobel edge detector to extract features from digital images of concrete surfaces and trained Neural Network and Fuzzy models to detect concrete cracks.

Chen et al. proposed a detection method for highway pavement damage [14]. The authors used a grayscale-weighted Histogram of Oriented Gradient feature patterns and a Convolutional Neural Network classifier.

 Jin et al. [15] used the Histogram of Oriented Gradients and the watershed algorithm to detect different pavement cracks and identify their severity. 

Another example was proposed by Zalama et al. [16]. The authors developed a methodology utilizing Gabor filters, several binary classifiers, and an Adaboost algorithm to detect longitudinal and transverse pavement cracks.

Chen et al. [17] presented a texture-based video processing approach using Local Binary Patterns (LBP), Support Vector Machine (SVM), and Bayesian decision theory to automate crack detection on metallic surfaces. Finally, Hu et al. [18] proposed a pavement crack detection method based on texture analysis using the Gray-Level Co-Occurrence Matrix, translation invariant shape descriptors, and an SVM classifier. 

Depending on the dimension of the extracted features, classifier training can be time-consuming and computationally expensive. This limitation can be addressed by applying dimensionality reduction techniques. For example, Principal Component Analysis, Linear Discriminant Analysis, and Isometric mapping can be used as a processing step in the general classification framework. In this sense, Abdelmawla et al.[19] applied image processing techniques, Principal Component Analysis (PCA), and K-means algorithm to detect and classify different types of cracks in pavement surface images. In addition, Abdel-Qader et al. [20] proposed an algorithm based on PCA, linear features, and local information to detect cracks in concrete bridge images. Kumar et al. [21] used PCA and a modified LeNet5 model to detect cracks in roads and bridges working with three different crack datasets. Endri et al. [22] used PCA as a preprocessing step to extract features from a pavement crack dataset containing 400 images. Chen et al. [23] extracted features from frames of road videos using LBP, reduced the dimension of the LBP feature space by PCA, and trained an SVM classifier to detect different types of pavement cracks. Elhariri et al. [24] proposed a crack detection model for historical buildings images based on feature extraction, a fusion of handcrafted features (i.e., HOG and LBP features) and Convolutional Neural Network-learned features, dimensionality reduction using PCA and Machine Learning classifiers. 

Comment 5: Overall comment: The results presented in this work needs a major improvement and I request you to improve it considerably.

 Authors response: We very much appreciate the reviewer’s comment. We have made the necessary changes in the manuscript.

Authors actions:  Dear reviewer, we are very grateful for your comment. We have conducted further experiments to improve the results of our work.

Kindly refer to our updated manuscript

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

The author made a detailed reply to the comments given last time, which shows the author's serious attitude.

However, I still reserve my original opinions on the novelty and frontier of the method employed in the manuscript. The difficulty of the algorithm employed in this paper is a bit too simple, and it is not suitable for publishing in the journals at this level. I still adhere to the original opinion and hope and encourage the author to replace the old method with the cutting-edge method.

Author Response

REVIEWER #2

Dear reviewer, Thanks a lot for your appreciated comment. We have carefully considered the reviewer suggestion and we hope that the reviewer finds our response acceptable. 

Comment 1: The author made a detailed reply to the comments given last time, which shows the author's serious attitude. However, I still reserve my original opinions on the novelty and frontier of the method employed in the manuscript. The difficulty of the algorithm employed in this paper is a bit too simple, and it is not suitable for publishing in the journals at this level. I still adhere to the original opinion and hope and encourage the author to replace the old method with the cutting-edge method.

Authors response: We very much appreciate the Reviewer’s comment.

Authors actions: Dear reviewer, we are very grateful for your suggestion. We have conducted several experiments using HOG and ULBP features as inputs to the t-Distributed Stochastic Embedding (t-SNE) for dimensionality reduction and the best classification results obtained using SVM are the following: 

HOG features

Accuracy: 66.34%

              precision    recall  f1-score   support

         0       0.65      0.85      0.74       513

         1       0.70      0.44      0.54       420

    accuracy                           0.66       933

   macro avg       0.68      0.64      0.64       933

weighted avg       0.67      0.66      0.65       933

ULBP features:

Accuracy: 62.91%

              precision    recall  f1-score   support

         0       0.61      0.93      0.73       513

         1       0.76      0.26      0.38       420

    accuracy                           0.63       933

   macro avg       0.68      0.60      0.56       933

weighted avg       0.68      0.63      0.58       933

In our proposed framework, dimensionality reduction using Kernel PCA yielded better results. This can be attributed to the limited number of features for t-SNE (3 features). Therefore, we added in the updated version of the paper three visualization maps of reduced HOG and ULBP features after the KPCA application using the Uniform Manifold Approximation and Projection (UMAP) algorithm. The resulting maps show that the samples are separable after the KPCA application.

We have added the following paragraph

To further visualize the HOG and ULBP features after the KPCA application, the authors employed the Uniform Manifold Approximation and Projection (UMAP) algorithm [27], which is a manifold learning and dimension reduction algorithm effective for cluster visualization.

Figure 6 presents the UMAP visualization maps of HOG and ULBP features after the KPCA application. It can be noticed that the samples represented by the reduced HOG and ULBP features can be separable, which is convenient for the classification models to differentiate cracked and uncracked concrete images.

                                                           (a)                                                                         (b)

                 Fig. 6. UMAP visualization maps of HOG and ULBP features after KPCA application: (a) reduced HOG features and (b) reduced ULBP features

Figure 7 presents the UMAP visualization map of HOG and ULBP features after KPCA application and feature fusion. It can be seen that the fusion of the reduced HOG and ULBP features resulted in a better separability of the studied samples and would presumably improve the performance of the classification models.

                 Fig. 7 UMAP visualization map of HOG and ULBP features after KPCA application and feature fusion

REVIEWER #3

Comment 1: Dear Authors,

The revised version of the paper is fine.

All my suggestions are addressed.

Authors response: We would like to thank you for the careful and thorough reading of this manuscript and the thoughtful and supportive comments and constructive suggestions, which help improve this manuscript's quality.

Author Response File: Author Response.pdf

Reviewer 3 Report

Dear Authors,

The revised version of the paper is fine.

All my suggestions are addressed.

Author Response

REVIEWER #3

Comment 1: Dear Authors,

The revised version of the paper is fine.

All my suggestions are addressed.

Authors response: We would like to thank you for the careful and thorough reading of this manuscript and the thoughtful and supportive comments and constructive suggestions, which help improve this manuscript's quality.

Author Response File: Author Response.pdf

Round 3

Reviewer 2 Report

The author made another round of revision according to the review comments, and the problems raised were basically explained or solved. The manuscript can published