An SVM-Based Scheme for Automatic Identification of Architectural Line Features and Cracks

Abstract

This research investigates fundamental problems in object recognition in earthen heritage and addresses the possibility of an automatic crack detection method for rammed earth images. We propose and validate a straightforward support vector machine (SVM)-based bidirectional morphological approach to automatically generate crack and texture line maps through transforming a surface image into an intermediate representation. Rather than relying on the application of the eight connectivity rule to a combination of horizontal and vertical gradient to extract edges, we instruct an edge classifier in the form of a support vector machine from features computed on each direction separately. The model couples a bidirectional local gradient and geometrical characteristics. It constitutes of four elements: (1) bidirectional edge maps; (2) bidirectional equivalent connected component maps; (3) SVM-based classifier and (4) crack and architectural line feature map generation. Relevant details are discussed in each part. Finally, the efficiency of the proposed algorithm is verified in a set of simulations that is satisfactorily conforming to labeled data provided manually for surface images of earthen heritage.

1. Introduction

Earthen structures are essential elements of world heritage. Among the most widely documented structural types, rammed earth is significantly spread. Conservation and restoration of earthen cultural heritage are firmly bound to the identification and characterization of existing cracks, which provide accurate quantitative data upon which conservation strategies are based. Severe cracks may propagate along the surface of an earthen wall, which may lead to severe structural problems and failure. This fact highlights the need to point out crack regions and manage surface cracks from their early stage from the safety point of view, which is fundamental for preservation purposes. This is the reason why it’s crucial to provide a crack detection algorithm that matches rammed earth images at the best of our possibilities.

In order to build a rammed earth wall, soil mixture is poured into a rigid formwork and compacted in layers. The wall is built a section at a time, and the formwork is moved from section to section, horizontally and vertically, until the full wall height is reached. The formworks for rammed earth are usually made of two panels and several long bolts in the middle to extend through the form and hold panels together. Once the wall is completed, the panels are removed, and the bolts are driven out of the wall. At the removal of the formwork, some characteristic joints are left in the wall, which are often seen as a feature of the rammed earth architecture, as shown in Figure 1.

Even though rammed earth is one of the oldest construction materials, there are still fundamental questions about the structure’s performance for researchers and engineers to solve. A large number of image processing techniques have been previously developed for detecting surface cracks from digital images for various purposes [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30]. These techniques, however, can only be applied to specific types of images and purposes. In particular, crack identification and characterization in rammed earth heritage is a topic not yet addressed in the literature. This study, carried out in the framework of Tech4Culture project, addresses this issue by studying and developing an automatic crack detection methodology for surface images of rammed earth buildings.

The existing automatic image-based techniques present some limitations for analyzing rammed earth images. Above all, detection results of existing image processing techniques are strongly influenced by the presence of architectural features, which may be misidentified as cracks. Automatic and accurate detection of cracks in the surface images of earthen structures is hence a challenging task.

This research attempts to propose an effective automatic crack detection algorithm focused on rammed earth, by exploiting the strong feature learning capabilities of learning-based approaches. We developed an automatic technique that allows the accurate generation of a crack map and architectural feature map that can describe the real condition of the structure, and thus avoid automatic misidentification.

Machine learning seeks to enhance the ability of computers to understand the real world through the creation of methods and techniques to obtain and process high-level information from images. Even though machines have seen much success in handling huge amounts of data with ease that humans have failed to achieve, the task of obtaining high-level information from an image turns out to be complex. In fact, it is often not sufficient to implement the machine learning algorithms by merely regarding the raw images as training examples. Instead, there are geometrically coherent fundamental elements in images that should be exploited in conjunction with learning techniques. This paper concentrates on the development of a novel support vector machine (SVM)-based approach for solving a real-world pattern recognition problem by transforming a surface image into an intermediate representation, which in turn can provide a set of mathematical rules for desirable tasks such as crack detection.

Our approach to crack detection couples a bidirectional local gradient and geometrical cues to a robust framework by using Connected Components Analysis (CCA) [31]. Rather than relying on the application of the eight connectivity rule to a combination of the horizontal and vertical gradient to extract edges, we instruct an edge classifier in the form of a support vector machine (SVM) [32] from features computed in each direction separately. The local cues, computed by employing oriented gradient operators, enable the determination of crack and architectural feature characteristics. From this, we derive a generalized feature selection problem and solve for surface images that encode edge information. Using a classifier to recombine these fundamental features with the local-oriented cues, we obtain a model for identification and labeling of both crack and architectural line features for surface images of rammed earth heritage.

We make two distinct contributions. First of all, we provide a labeled dataset of geometrical characteristics of crack and architectural line features for training and evaluating the performance of the proposed algorithm on rammed earth images. Secondly, we propose an algorithm for the automatic generation of a crack map and an architectural feature map for rammed earth images. Given the importance of crack detection in the conservation of earthen heritage, we believe these data and this algorithm will help advances in research in the field.

2. Developed Methodology

The algorithm is developed on the Matlab 2018a. The basic building block of the established procedure is the computation of an oriented gradient in horizontal and vertical directions at every pixel in an image (Figure 2). This computation proceeds by CCA [31], which is equivalent to fitting an ellipse having the same second-moments as the connected component (CC), whose major axis is oriented along direction θ (Figure 3). The properties of cracks and non-cracks at a location (x, y) are equivalent to the geometrical properties estimated by the fit. An example of such an equivalent image can be seen in Figure 3.

After constructing CC maps in horizontal and vertical directions, the algorithm proceeds by the analysis of connected components, whose area is more than 30 pixels, from the largest to the smallest ones. For each CC, the algorithm classifies it as a crack or architectural feature using an SVM-based classification scheme. Then, it constructs the crack map and the architectural feature map by adding up the corresponding horizontal and vertical derivatives of pixel intensities followed by binarization using Otsu’s method [33]. This entire procedure is repeated until all pixels belonging to horizontal and vertical CCs are analyzed once.

2.1. Constructing Edge Maps

As the first step in this process, the input image is converted to grayscale, and oriented edge maps are constructed by applying edge detection operators in horizontal and vertical directions independently (Figure 2). This section presents a review of edge detection algorithms and reports the results of performance evaluation in rammed earth images for this task. The purpose of this study is to evaluate the capability of these methods to separate crack objects from the background. This framework discusses two aspects of detection performance, including performance in crack detection and performance in the identification of architectural features. Shortcomings of existing techniques and relevant details are taken into account. Such an analytical framework is necessary for the selection of an edge detection technique for the problem at hand. Focus is particularly given to find solutions that overcome limitations. The main requirement that is expected from the solution is a reasonably good accuracy in the automatic identification of the presence of multiple cracks and architectural objects in a rammed earth image. Motivated by the observed inconsistencies and inaccuracies to distinguish between cracks and architectural features, this section concentrates on the selection of an edge detection technique, whose main algorithm is inspired by the performance analysis.

A broad family of approaches to crack detection relies on modeling cracks as sharp discontinuities in the brightness, which is a characteristic of our approach as well. While this is by no means the only path taken, local edge detection operators appear to be the most widely used.

The edge detection method is a simple but effective tool to identify and locate the pixels that correspond to the boundaries of an object in the image, which can be obtained by mask processing techniques [34]. The output of this operation is a binary image with the detected edge pixels. However, several of those pixels may be the result of noise or surface features. Early local approaches, such as the Sobel operator [35], quantifies the presence of a boundary through discontinuity measurements in the local brightness. Another family of methods, such as wavelet transform, aims at representing the image through a collection of sub-bands, more basic images containing different fine-scale and large-scale information of the original image. An example is fast Haar transform (FHT) [36], which produces essentially three directional edge maps, namely horizontal, vertical and diagonal. The most commonly used edge detection algorithms are Sobel [35], Canny [37], Prewitt [38], Laplacian of Gaussian (LoG) [39] and FHT [36]. In this section, these algorithms are applied to extract crack pixels from selected parts of rammed earth images. Depending on the application, these algorithms may produce missing or extra edges on complex surface images. Investigating these algorithms can help in the choice of an edge detection technique over another for automatic crack detection of rammed earth heritage.

Edge detection methods usually determine the sharp variation in the gray level intensities based on the calculation of the first and second derivatives of the local intensities. The first and second derivatives of an image are calculated by the gradient vector and Laplacian Operator. The Sobel [35] edge operator provides vertical and horizontal edges defined by two horizontal and vertical masks. Like Sobel [35], Prewitt [38] operator performs a two-dimensional spatial gradient measurement, which consists of a pair of 3 × 3 convolution kernels. One kernel estimates gradient in the x-direction and the other estimates gradient in y-direction. The LoG operator [39] has been developed based on the calculation of the second derivative of the local intensities. In this operator, the sensitivity of Laplacian operator to noise has been reduced by operating Gaussian filter first.

Haar transform [36] of matrices computes averages and differences of neighboring pixels in various combinations, which can be used for identifying and locating edges in an image. Haar transform [36] of an image generates a collection of sub-bands, more basic images; each of them contains different fine-scale and large-scale information of the original image. Multi-scale features of the original image can be extracted directly from the Haar decomposition coefficients, namely detail and approximation coefficients.

Experimental results obtained at this stage describe the performance of the above-mentioned techniques in practical crack detection in rammed earth heritage. Figure 4 and Figure 5 show some representative results of the edge detection analysis.

As observed through this experimental study (Figure 4), the Sobel edge operator performs better in the extraction of crack pixels from the background for rammed earth images. In particular, Sobel predicts fewer missing pixels, although this operator may get a noisier output. Another observation confirms that the implementation of crack detection algorithms cannot achieve optimal performance by merely applying an edge detection operator on a rammed earth image. This can be attributed mainly to the fact that the detection results of the algorithms are influenced by architectural objects. Therefore, these methods are not appropriate for complex images containing architectural features. As observed in Figure 6, edge detection algorithms often join horizontal and vertical architectural lines at the corners (Figure 6a) or architectural lines and cracks at the intersections (Figure 6b,c). To circumvent this difficulty, we apply Sobel edge detection operators in the horizontal and vertical direction separately (Figure 2). Taking derivatives in this manner confirms that oriented edge maps themselves carry high-level feature information.

Moreover, it is observed that Haar transform method can be an appropriate way to detect horizontal and vertical line objects. Figure 6 shows representative results for the Haar transform implemented on a rammed earth image containing horizontal and vertical architectural lines. It can be concluded that this method can identify horizontal and vertical joints in a rammed earth image and categorize and map them as architectural features. Although a feature classification can be obtained by representing an image through three directional edge maps at different scales and orientations, in rammed earth structures containing multiple cracks and architectural line features, this can lead to an incorrect classification, as directed features of an edge are broken up and mapped in different sub-band images. This means that horizontal sub-band does not contain horizontal lines. Rather, it includes horizontal features of all lines in an image. From this, we found that constructing a simple directed edge map is not enough to benefit from feature classification. We solve this limitation by integrating local intensities and connectivity in each direction and then classifying those directed connected components based on geometrical features.

Treating each oriented edge map as an image, we apply CCA for each direction, obtaining equivalent bounding ellipses. This solves the problem of misidentification of CCs at the intersection of horizontal and vertical lines or architectural lines and cracks that previously lead to errors. The information from different horizontal and vertical edge maps provides far superior input for high-level task of crack and non-crack classification, which is then combined to provide an architectural line map and crack map. Figure 7 breaks down the contributions of oriented edge maps to the performance of the algorithm.

2.2. Connected Component Analysis

Once an input image was transformed into two separate edge maps using Sobel operators, each map is treated as an image and is processed independently. As previously observed in Figure 6, a CC built up from detected edges will often mistakenly join architectural lines and cracks, resulting in misidentification of geometrical properties. Hence, value exists in producing oriented gradient maps rather than single-level edge maps. This computation is motivated by the intuition that horizontal and vertical lines, left on the surface of the rammed earth structures, correspond to horizontal and vertical discontinuities in intensity and geometrical features provide a robust mechanism for modeling the content of such image regions. By using oriented edge detection output, CC regions are built by exploiting the information in these edge maps by applying the eight connectivity rule [31] to each group of neighboring pixels in each direction, independently (Figure 7).

Since cracks differ from architectural lines considering physical properties, their geometric characteristics can be used to classify them into different classes. Therefore, various morphological parameters can be defined to group them mathematically. For example, a strongly oriented axis length ratio (Ratio of Major Axis Length to Minor Axis Length) means a CC is more likely to lie on an architectural line map. In the previous stage, crack and non-crack regions were extracted from images using a classical edge detection technique, Sobel edge operators, in horizontal and vertical directions. To classify crack and architectural features, inspecting the detected oriented edge maps pixel by pixel is not necessary. In this model, we start from CC, a group of neighboring pixels that form a roughly linear segment. Each CC is represented by an ellipse. CC fundamental and physical properties are equivalent to those of this ellipse, and are utilized to support subsequent automatic crack analysis.

A wide range of a surface can be acquired with a digital camera [40]. Therefore, fundamental analyses are performed on patches with 256 × 256 pixels. Moreover, by expressing the geometrical characteristics in terms of the normalized quantities, we obtain a scale-invariant feature space. The morphological parameters identified and selected to use for CC analysis are briefly described in

Table 1. Morphological parameters for connected component analysis.

Area	Number of Pixels
Density	Ratio of pixels in the region to pixels in total image
Major axis length	Length (in pixels) of the major axis of the equivalent ellipse
Minor axis length	Length (in pixels) of the minor axis of the equivalent ellipse
Length ratio	Ratio of the major axis length to 256√2
Width ratio	Ratio of the minor axis length to 256√2
Axis length ratio	Ratio of Major Axis Length to Minor axis length
Orientation	Absolute value of the angle between the x-axis and the major axis of the ellipse, ranging from 0 degrees to 90 degrees (Figure 3b).
Orientation ratio	Ratio of orientation to 90 degrees
Convex area	Number of pixels in convex Image (Figure 8c)
Convex area ratio	Ratio of pixels in convex area to pixels in total image
Equivalent diameter	Diameter of an equivalent circle with the same area as the region
Equivalent diameter ratio	Ratio of Equivalent diameter to 256
Extent	Ratio of pixels in the region to pixels in the total bounding box
Solidity	Ratio of area to convex area

and shown in Figure 8. We assign to each CC on each oriented map the morphological parameters of the corresponding ellipse. The results of CC analysis are shown in Figure 9.

2.3. SVM-Based Classification Scheme

Once a set of features was selected and measured, dimensions are reduced to three fundamental features. Applying the results of CCA, allows us to represent data in lower dimensions without compromising the meaning of the data. Figure 10a depicts further visualization of well-separated data in three-dimensional (3D) feature space. These geometric feature vectors, axis length ratio, orientation and length, are used as input to train an SVM [32]. This is motivated by the intuition that monitoring systems and analytical approaches take into account this geometrical information and make use of crack length and width as fundamental parameters for determining the severity of cracks. Moreover, in this case, these three parameters yield a high degree of variance, according to Figure 9. Reduction in computational complexity for analytical systems as outlined above, as well as a high degree of separability for these three parameters, make our crack detector methodology a practical tool. As one of the most widely used machine learning techniques, SVM [32] methodology performs classification by finding the maximum-margin hyperplane that separates two different categories with the best performance and maximizes the margin between the nearest data points and the hyperplane in different classes.

The Simple SVM uses binary learners to train the model. However, this type of model tends to suffer from over-fitting. To solve this problem, cross-validation techniques are employed herein. In cross-validation techniques, a subset of data called “validation set” is used to test the classifier during the training stage. In an n-fold cross-validation technique (Figure 11), such as five-fold cross-validation [41], first, the training set is divided into n subsets equally. In order to ensure the proper learning, the mechanism is to test one subset using the model trained on the remaining n−1 subsets. This process is repeated n times so that each instance of the whole training set is validated once. The accuracy of the model is computed based on the average accuracy of n classifiers [41].

In order to classify crack and architectural line features, a cross-validated SVM is applied to train a binary model, exploiting a three-dimensional feature space (Figure 10). In total, 80% and 20% of data are considered as training and test data, respectively. To ensure proper learning, five-fold cross-validation is performed. The classifier achieves an accuracy of 98.10%.

2.4. Map Generation

We then utilize the output of the classifier as well as associated horizontal and vertical edge pixel intensities for each CC to construct crack and architectural feature maps. In order to do this, the algorithm considers all CCs with areas more than 30 pixels, in horizontal and vertical CC maps, from largest to smallest one. Once each CC is classified as crack or architectural feature, the corresponding pixel information from oriented edge maps is extracted and then combined to generate desirable feature maps. Crack and architectural line maps are generated by adding up associated pixel intensities in horizontal and vertical edge maps. This entire procedure is repeated for all CCs in such a way that each pixel of the whole CCs is analyzed once. As a consequence, the output is represented as the ultimate crack map and architectural feature map (Figure 2).

3. Model Evaluation

To evaluate the performance of the algorithm, we report two important results. First, we conduct some initial experiments in an attempt to determine the performance of the classifier on a new set of data from rammed earth heritage images. These data have not been used in training and validating the classifier. Figure 12 presents the data used for this experiment.

We also analyze the sensitivity of the accuracy of the developed model to variations in three fundamental parameters for identifying cracks and architectural lines (Figure 13). According to this analysis following observations can be made:

-

For a specific orientation, the boundary value of the axis length ratio, the maximum value of the axis length ratio for an equivalent ellipse to be considered as a crack, increases with an increase in crack length. This highlights the high accuracy of the model in the identification of severe and thin cracks for any directions rather than horizontal and vertical ones, which can be indications of structural crack initiation.

-

When orientation approaches horizontal and vertical directions, the boundary value of axis length ratio decreases for a specific length of the feature. This justifies the accuracy of the developed model in detecting the presence of very thin architectural lines left on the walls due to the construction technology of rammed earth buildings. On the other hand, this confirms the ability of the model to detect severe cracks in horizontal and vertical directions.

Then, the developed algorithm is evaluated against experimental data for producing crack map. We evaluate performance with respect to the crack and non-crack labels provided manually for equivalent ellipses. Figure 14, Figure 15, Figure 16 and Figure 17 illustrates the results of the evaluation of the algorithm for producing the crack map from images of rammed earth heritage as an input. Note that of the 287 regions examined for classifier evaluation, 98.26% is reported as true positive and false negative. False positives (0.70%) are concentrated on very thin cracks, whose misclassification as architectural features is still acceptable, since these cracks usually represent texture cracks rather than structural ones. The results of this study also indicate that true negatives (1.04%) may be observed on rammed earth images with very specific texture characteristics.

4. Conclusions

With special attention to morphological characteristics of cracks and architectural line features in surface images of earthen heritage, the current study proposes a scale-invariant SVM-based framework whose main algorithm associates with each object morphological characteristics of bidirectional bounding ellipses, namely axis length ratio, orientation ratio and length ratio, and uses an SVM classifier to create crack and architectural line feature maps. Rather than relying on the application of the eight connectivity rule to the combination of horizontal and vertical gradient to extract edges, features are computed in each direction separately.

It turns out that equivalent ellipse, representing a connected component, is a sufficiently good approximation for our purpose. Inclusion of three directed morphological characteristics, axis length ratio, orientation ratio and length ratio, contributes positively to the performance of the crack detection algorithm for rammed earth images. The algorithm performs well on complex images, which contains a combination of different cracks and architectural line features. This approach is quite accurate at the classification of horizontal texture lines, the characteristics of most rammed earth walls. Another advantage of this approach is its ability to distinguish severe vertical cracks, one of the most common cracks in rammed earth heritage, and vertical architectural lines. The results indicate a high degree of accuracy in producing crack and architectural feature maps.

The accuracy with very thin cracks is slightly lower than with the wider ones. However, their misclassifications as architectural line features are still acceptable, since these cracks usually represent texture cracks rather than structural ones. The results of this study also indicate that the algorithm adds some true negatives in rammed earth structures with particular texture characteristics. The authors are currently addressing this issue by investigating a different methodology, aiming at overcoming this problem.