Computer-Vision and Machine-Learning-Based Seismic Damage Assessment of Reinforced Concrete Structures

Abstract

Seismic damage assessment of reinforced concrete (RC) structures is a vital issue for post-earthquake evaluation. Conventional onsite inspection depends greatly on subjective judgments and engineering experiences of human inspectors, and the efficiency is limited to large-scale urban areas. This study proposes a computer-vision and machine-learning-based seismic damage assessment framework for RC structures. A refined Park-Ang model is built to express the coupled effects of structural ductility and energy dissipation, which reflects the nonlinear seismic damage accumulation and generates a synthetical seismic damage indicator within 0~1 using hysteretic curve data. A deep neural network is established to regress the damage indicator using damage-related and design-related parameters as inputs. The results show that the correlation coefficients between the predicted and actual seismic damage index exceed 0.98, and the predicted seismic damage index is unbiased and stable without overfitting. Furthermore, the effectiveness, robustness, and generalization ability of the proposed method are verified.

1. Introduction

The seismic damage assessment of reinforced concrete (RC) structures is a vital issue for the post-earthquake evaluation of building structures, especially for large-scale urban areas. Therefore, quantitative seismic damage assessment using damage models is essential to expound on the failure mechanism of earthquake-induced damage for RC structures. Park and Ang [1] first proposed an evaluation model for the seismic damage of RC structures using a linear function of the maximum deformation and the effect of repeated cyclic loading. Since then, researchers have dedicated themselves to developing novel evaluation models for seismic damage [2,3], and several response-based damage indexes have been summarized and investigated for actual applications of seismic damage evaluation [4]. Later, Kim et al. [5] integrated a nonlinear finite element analysis procedure with a damage index calculated using hysteretic behaviors for the seismic damage assessment of RC columns. In addition, the damage model also considered low-cycle fatigue and repeated loading severity [6]. As for modifying the original Park-Ang model, a modified model was proposed by eliminating its non-normalization issue [7]. Multivariable nonlinear regression analysis was performed to obtain the empirical formula of the combination coefficient from the design parameters. A generalized multi-level deformation-based continuum damage model with stiffness degradation was developed to evaluate the seismic damage of RC-framed structures [8]. A nonlinear damage model considering the mutual effect of deformation and hysteretic energy was designed for different types of components in different loading stages [9]. Lin et al. [10] established a seismic damage model for bridge piers subjected to biaxial loading considering the impact of energy dissipation. Wang et al. [11] designed a damage index for RC members based on energy dissipation capability degradation. Besides RC structures, seismic damage assessment has also been performed on recycled concrete columns and steel beam composite frame joints to analyze the failure modes and ductility [12]. A series of reviews have summarized the research progress of seismic damage indices for RC structures [13,14,15,16].

Recently, the rapid development of artificial intelligence has significantly promoted the evolution of earthquake engineering [17,18,19,20], especially the broad applications of a variety of machine-learning techniques [21,22,23,24], e.g., artificial neural networks [25], machine vision [26], decision trees [27], support vector machines [28], and convolutional neural networks [29,30]. For example, Luo and Paal [31] established a locally weighted machine-learning model for the generalized prediction of drift capacity in seismic vulnerability assessments. Mangalathu and Jeon [32] successfully recognized the failure mode of circular RC bridge columns using machine learning. Lu et al. [33] proposed a deep-learning approach to rapid regional post-event seismic damage assessment using time-frequency distributions of ground motions.

As one of the most significant branches of artificial intelligence, computer vision integrated with remote sensing [34] and unmanned aerial vehicle (UAV) [35] techniques have been widely applied in earthquake engineering. A series of investigations have been performed to detect multi-type seismic damage [36,37], measure structural dynamic response under earthquakes [38], reconstruct the three-dimensional scenario of city-scale buildings [39,40], and assist post-earthquake inspections [41] based on experimental images [42,43]. The related advances have been comprehensively summarized in recent review papers [44,45,46].

For example, rapid building localization and binary classification of collapse or non-collapse for small, dense buildings in broad areas were achieved using a modified You Only Look Once (YOLO) v4 and support vector machine [47]. Shao et al. [48] proposed a novel end-to-end remote-sensing pixel-classification deep convolutional neural network for classifying non-damaged buildings, damaged buildings, and backgrounds. Xiong et al. [49] utilized geographic information system data and projection transformations to obtain segmentation images of individual buildings from tilting photography taken by UAVs and built a convolutional neural network (CNN) to assess whether buildings had collapsed. Kakooei and Baleghi [50] reported an automatic fusion framework for building damage assessment by combining satellite and UAV images. Duarte et al. [51] designed a CNN framework using residual connections and dilated convolutions to combine satellite and airborne images and to improve the classification accuracy of damaged buildings.

In addition to vision-based seismic damage recognition, Hoskere et al. [52] and Narazaki et al. [53] proposed a framework to generate vision-based condition-aware models using UAV images and integrated the geometry information of structural components, structural defects, and damage states into a three-dimensional model. Dogan et al. [54] proposed an intelligent system-oriented method to determine the damage levels of RC columns. Feng et al. [55] explored an ensemble machine-learning algorithm to classify failure modes and predict the bearing capacity for RC columns. Mangalathu et al. [56] proposed a SHapley Additive ex-Planations approach to analyze the failure mode of RC members and further performed rapid structural damage assessment based on wavelet transform and image analysis techniques [57]. Liu et al. [58] designed an information fusion strategy to automatically classify post-event building damage states.

It should be noted that images acquired by different platforms have unique advantages and characteristics. The satellite remote sensing images can quickly obtain the large-scale general location of building groups, but only the visible information from building roofs affects the evaluation accuracy. UAV images can obtain much more precise information about building facades, but the inspection range is limited due to the power endurance. Although a multi-scale damage recognition, localization, and assessment framework can be established for post-earthquake building evaluation, it is based only on computer-vision techniques. It is meant to build mapping relationships between seismic images in various scales and the corresponding damage levels end-to-end. Almost all established recognition models lack intrinsic connections with structural mechanics and physical properties and are regarded as black boxes. Therefore, the model interpretability of pure computer-vision-based seismic assessment faces significant challenges.

To overcome the abovementioned issues, this study proposes a framework based on computer-vision and machine-learning techniques to quantify the seismic damage of RC structures. A refined Park-Ang model is built using the maximum deformation and cumulative energy consumption based on hysteresis curves to generate a synthetical seismic damage indicator within 0~1. Subsequently, a deep neural network is established to regress the seismic damage indicator using multi-type seismic damage and structural design variables, including the total relative lengths of concrete cracks in horizontal and vertical directions, the total number of concrete cracks, the relative area of concrete spalling, the relative area of rebar exposure, the shear span ratio, the axial compression ratio, and the volumetric stirrup ratio. The damage-related variables are directly derived from images using a computer-vision-based region detection model for multi-type seismic damage to RC structures, and the design-related variables are obtained from the corresponding design drawings. Both seismic-damage-related parameters and structural-design-related parameters are utilized as inputs. Finally, an autonomous fuzzy evaluation strategy is proposed for five-level seismic damage classification of real-world post-earthquake inspection images without structural design parameters. The necessity, effectiveness, robustness, and generalization ability of the proposed method are further demonstrated and verified through a series of additional comparative studies and real-world applications.

The remainder of this paper is organized as follows. Section 2 introduces the overall schematic, refined Park-Ang model, and deep neural network for the seismic damage assessment of RC columns. Section 3 presents the implementation details, including the investigated dataset and optimization of the network structure and training hyperparameters. Section 4 demonstrates the efficacy and necessity of the refined Park-Ang model, shows the training and validation results of the seismic damage index prediction, performs statistical analysis of regression errors, and verifies the robustness and generalization ability of the proposed method for real-world applications. Finally, Section 5 concludes the paper.

2. Methodology

2.1. Computer-Vision and Machine-Learning-Based Framework for Seismic Damage Assessment

This study establishes a seismic damage assessment framework using computer vision and machine learning for RC structures based on seismic inspection images, structural design information, and quasi-static experimental data. Specifically, seismic damage-related variables, including horizontal and vertical crack lengths, number of concrete cracks, and areas of concrete spalling and rebar exposure, are recognized from seismic inspection images using a computer-vision-based region detection model for multi-type seismic damage. The design-related variables, including the shear span ratio, axial compression ratio, and volumetric stirrup ratio, are directly obtained from the design drawings. Then, a refined Park-Ang model is built to calculate a synthetical seismic damage indicator with an explicit bound of [0, 1], which acts as the ground truth of seismic damage severity. Finally, a multi-layer neural network is trained to regress the seismic damage indicator using seismic damage-related and design-related variables as the inputs. Figure 1 shows the overall schematic of the proposed machine-learning framework for the seismic damage assessment of RC structures.

Note that a deep neural network is established for the damage index regression from the input variables of both damage-related parameters (sizes and numbers of various damage regions of concrete crack, concrete spalling, and rebar exposure) and structural design parameters (axial compression ratio, shear span ratio, and volumetric stirrup ratio). The damage-related parameters are automatically derived from images using a region-based object detection model for multi-type seismic damage by Xu et al. [36].

2.2. Refined Park-Ang Model Considering Nonlinear Accumulation of Seismic Damage

The classical Park-Ang model [1] was proposed based on structural deformation and energy dissipation for quantitative damage assessment under seismic loads, as follows

$$D_{\mathrm{original}}=\frac{{\delta }_m}{{\delta }_u}+\beta \frac{\int dE}{F_y{\delta }_u},\beta =(-0.447+0.73\lambda +0.24n_0+0.314{\rho }_t)\times 0.7{\rho }_v$$

(1)

where $D_{\mathrm{original}}$ is the original seismic damage index proposed by the conventional Park-Ang model [1]; ${\delta }_m$ and ${\delta }_u$ are maximum deformations under cyclic and monotonic loads, respectively; $\int dE$ is the cumulative energy consumption calculated by the hysteresis curves; $F_y$ is the yielding force; $\beta$ is the coefficient between the deformation-related term and the energy-related term, which is empirically determined; $\lambda$ is the shear span ratio; $n_0$ is the axial compression ratio; ${\rho }_t$ is the reinforcement ratio; and ${\rho }_v$ is the volumetric stirrup ratio.

The physical basis of the conventional Park-Ang model is that the seismic damage of the structural components is induced by the coupled effects of the maximum structural deformation and cumulative energy dissipation under cyclic seismic loads. For RC structures, concrete cracks as the primary damage propagate slowly and take a long time at the early stage of quasi-static experiments, while the corresponding damage index should remain at a low level and increase gradually. Later on, the ductility and stiffness of RC components decrease significantly after severe damage (crack spalling and rebar exposure) occurs, corresponding to a high and rapidly-increasing damage index. Previous studies [1,7,11] have shown that the coefficient $\beta$ for most RC components is often beneath 0.1, which inevitably underestimates the effect of nonlinear damage accumulation under cyclic seismic load and brings an intrinsic error for seismic damage assessment. Moreover, the deformation-related term is determined by the ratio of maximum deformations under cyclic and monotonic loads, which cannot remain zero during the elastic stage.

Therefore, the conventional Park-Ang model has the following limitations: (1) the upper and lower bounds of the damage index for different RC components are not explicit; (2) the damage index is larger than 0 during the elastic stage, reaches a high value in the early stage, and exceeds 1 when the final failure occurs; and (3) the nonlinear expediting phenomenon of seismic damage accumulation cannot be accurately reflected.

To solve the above limitations, a refined Park-Ang model is proposed in this study as

$$D=(1-\beta )\frac{u_r}{u_y}+\beta \frac{\int dE}{F_y{\delta }_m}$$

(2)

where D is the refined seismic damage index, $u_r$ is the residual deformation, $u_y$ is the yielding deformation, and ${\delta }_m$ is the maximum deformation under cyclic loads. $\int dE$, $F_y$, and $\beta$ are defined in Equation (1).

Based on the quasi-static experimental data of RC columns, the skeleton curve can be obtained from the hysteresis curve, as shown in Figure 2a,b. The yielding deformation and yielding force of the RC columns under cyclic loads are calculated as the average value in positive and negative cycles, as follows

$$F_y=\frac{F_y^{+}+\left|F_y^{-}\right|}{2},{\delta }_m=\frac{u_y^{+}+\left|u_y^{-}\right|}{2}$$

(3)

where $F_y^{+},F_y^{-}$ are the equivalent yielding forces in positive and negative cycles of the skeleton curve, respectively, and $u_y^{+},u_y^{-}$ are the equivalent yielding deformations in positive and negative cycles of the skeleton curve, respectively. The yielding force $F_y$ and yielding deformation $u_y$ can be determined by the equivalent energy method, as shown in Figure 2c. The yielding deformation $u_y$ is equal to that of the equivalent elastic-plastic system with the same energy dissipation capacity ($S_1=S_2$) under the identical maximum external load $F_{\max }$.

The coefficient $\beta$ in Equation (2) is determined by setting $D=1$ as

$$\beta =\frac{F_y{\delta }_m(u_y-u_r)}{u_y\int dE-F_y{\delta }_m{u}_r}$$

(4)

The refined Park-Ang model has the following properties: (1) the seismic damage index has an explicit bound of [0, 1] and remains zero in the elastic stage because no residual deformation and energy dissipation occur; (2) as the residual deformation is small in the early stage, a correction coefficient of $(1-\beta )$ is introduced to the deformation-related term; and (3) the nonlinear cumulative effect of seismic damage evolution is fully considered in the energy-related term.

2.3. Image-Based Autonomous Regression of the Seismic Damage Index Using the Deep Neural Network

The region-based multi-type seismic damage recognition method proposed by Xu et al. [36] was utilized in this study to classify and localize concrete cracks, concrete spalling, and rebar exposure from quasi-static experimental images of RC columns. After the object detection task of seismic damage regions was performed, the rectangular bounding boxes for concrete cracks, concrete spalling, and rebar exposure were generated. Based on the pre-labeled bounding boxes for the column, the relative lengths of the concrete cracks in two directions and the relative areas of concrete spalling and rebar exposure were calculated. Therefore, these damage-related parameters were extracted and utilized as partial inputs for the deep neural network to predict the seismic damage index in an end-to-end manner. As for the design-related parameters, the shear span ratio, axial compression ratio, and volumetric stirrup ratio were obtained from the design drawings and were not derived from the images.

As concrete cracks are supposed to increase during the entire process of the quasi-static experiment, the total lengths of concrete cracks in horizontal and vertical directions were two significant variables, which mainly reflected the crack propagation process in the early and late stages, respectively, and could be derived by

$$w_c=\frac{{\displaystyle \sum _{i=1}^n{w}_i^{crack\_box}}}{w^{column\_box}}\times W,h_c=\frac{{\displaystyle \sum _{i=1}^n{h}_i^{crack\_box}}}{h^{column\_box}}\times H$$

(5)

where $w_c,h_c$ are the total lengths of concrete cracks in horizontal and vertical directions, respectively; $W,H$ are the actual width and height of the RC column, respectively; $w_i^{crack\_box},h_i^{crack\_box}$ are the width and height of the bounding box for the ith crack (unit: pixel), respectively; $w^{column\_box},h^{column\_box}$ are the width and height of the bounding box for the RC column (unit: pixel), respectively; and n is the total number of concrete cracks in the investigated image.

To further consider the nonlinear accumulation effect in the late stage of seismic damage evolution, areas of concrete spalling and rebar exposure were derived as follows

$$A_{spall}=\frac{w^{spall\_box}\times h^{spall\_box}}{w^{column\_box}\times h^{column\_box}}\times W\times H,A_{rebar}=\frac{w^{rebar\_box}\times h^{rebar\_box}}{w^{column\_box}\times h^{column\_box}}\times W\times H$$

(6)

where $A_{spall},A_{rebar}$ are the areas of concrete spalling and rebar exposure, respectively; $w^{spall\_box},h^{spall\_box}$ are the width and height of the bounding box for concrete spalling (unit: pixel), respectively; and $w^{rebar\_box},h^{rebar\_box}$ are the width and height of the bounding box for rebar exposure, respectively (unit: pixel).

Note that this study did not distinguish rebar exposure and rebar buckling. Based on the recognition results of the region-based object detection model, all of the local regions of rebar exposure and buckling were combined into one type of rebar exposure. The fundamental perspective was that rebar buckling could be regarded as a progressive state of rebar exposure, and this assumption would be conservative for seismic damage assessment. In addition, for the actual application of real-world post-earthquake inspection, rebar exposure is a rather severe damage type because the failure of the concrete protective layer already occurs.

In addition to the abovementioned damage-related parameters, effects from different shear span ratios, axial compression ratios, and volumetric stirrup ratios as design-related parameters were also considered. For the RC columns investigated in this study, the reinforcement ratio, yielding strength of longitudinal rebar, concrete compressive strength, and section size were the same and thus were not considered as the input variables.

In this study, the relative lengths of the concrete cracks in the horizontal and vertical directions and relative areas of concrete spalling and rebar exposure were calculated as the relative ratio compared with the column height and width, according to Equations (5) and (6). However, considering that the columns were different in terms of their height and width, the column size itself was also utilized as an input parameter. Therefore, the relative lengths of the concrete cracks in the horizontal and vertical directions and the relative areas of concrete spalling and rebar exposure were fused together with the column sizes as the minimal simplified input parameters in order to simultaneously reduce the correlations between the input feature variables and to ensure the model generalizability.

Finally, eight variables extracted from the seismic image of an RC column were used as the inputs, including five damage-related parameters (the total relative lengths of concrete cracks in horizontal and vertical directions, total number of concrete cracks, relative area of concrete spalling, and relative area of rebar exposure) and three design-related parameters (shear span ratio, axial compression ratio, and volumetric stirrup ratio). A deep neural network was established to map the relationship between the input variables of the damage-related and design-related parameters to the seismic damage index calculated in Section 2.2. The overall schematic for the autonomous regression of the seismic damage index based on images and quasi-static experimental data is shown in Figure 3.

The deep neural network contains an input layer (five damage-related parameters and three design-related parameters), eight hidden layers, and an output layer (seismic damage index). The optimal numbers of hidden layers and neurons in each hidden layer were determined by grid search, which is described in detail in Section 3.2. The mathematical operations and functional layers in the deep neural network were omitted here to avoid duplication, as they are common knowledge in the community. An average regress loss with an additional L₂ regularization term was utilized as the loss function to optimize the deep neural network. The established neural network was optimized by the Adam algorithm according to Ref. [59] as follows

$$\begin{array}{l}L({\omega }^{(1)},{\omega }^{(2)}\cdots {\omega }^{(L)})=\frac{1}{2N}{\displaystyle \sum _{i=1}^N{\Vert D^i-{\widehat{D}}^i\Vert }^2}+\frac{1}{2}\alpha {\displaystyle \sum _{l=1}^L{\Vert {\omega }^{(l)}\Vert }_2^2},g_t={\nabla }_{\omega }L\\ m_t={\beta }_1{m}_{t-1}+(1-{\beta }_1)g_t,v_t={\beta }_2{v}_{t-1}+(1-{\beta }_2)g_t^2\\ {\widehat{m}}_t=\frac{m_t}{1-{\beta }_1{}^t},{\widehat{v}}_t=\frac{v_t}{1-{\beta }_1{}^t},{\omega }_t^{(l)}\leftarrow {\omega }_{t-1}^{(l)}-\eta \frac{{\widehat{m}}_t}{\sqrt{{\widehat{v}}_t}+\epsilon }\end{array}$$

(7)

where ${\omega }^{(l)}$ is the lth hidden layer in the established deep neural network; L is the total number of hidden layers; N is the batch size; $D^i,{\widehat{D}}^i$ are the predicted and actual seismic damage indexes for the ith sample, respectively; $\alpha$ is the L2 regularization hyperparameter; ${\nabla }_{\omega }L$ is the gradient of the loss function with respect to the network parameter; ${\beta }_1,{\beta }_2$ are the first-order and second-order attenuation rates, respectively; $m_t,v_t$ are the first-moment and second-moment estimations of the gradient for the current time step, respectively; ${\widehat{m}}_t,{\widehat{v}}_t$ are the exponential moving averages for bias correction; $\eta$ is the learning rate; and $\epsilon$ is the constant preventing a zero denominator.

3. Implementation Details

3.1. Dataset of Seismic Images and Quasi-Static Experimental Data

The investigated dataset contained the quasi-static experimental data and the corresponding seismic images for a total of 124 RC columns from PEER [60] and previous studies [36,61]. Multi-type seismic damage, including concrete cracks, concrete spalling, and rebar exposure, was recorded during the entire process of quasi-static experiments in multiple scenes, scales, and resolutions. Data from RC columns No. 1~No. 100 were used for training, and the rest of RC columns No. 101~No. 124 were used for validation. Figure 4 shows several representative seismic images and bounding box annotations of RC columns with multi-type seismic damage. This indicates that the severity distributions and geometrical features of the investigated seismic damage were quite different and that complex backgrounds of structural edges and other disturbances were included.

In addition, inputs of the established deep neural network were damage-related and design-related variables of seismic RC columns. Each dimension of the input variables corresponded to an individual physical property. Therefore, all of the input parameters were normalized by the corresponding channel-wise maximum and minimum values to stabilize the network optimization and speed up the final convergence, as follows

$$x=\frac{x-x_{\min }}{x_{\max }-x_{\min }}$$

(8)

where x represents all of the input channels for seismic damage index regression, which corresponds to five damage-related variables (total lengths of concrete cracks in horizontal and vertical directions, total number of concrete cracks, area of concrete spalling, and area of rebar exposure) and three design-related variables (shear span ratio, axial compression ratio, and volumetric stirrup ratio). Subscripts of “max” and “min” represent the maximum and minimum values for the considered parameters in the dataset, respectively.

3.2. Optimization of Network Structure and Training Hyperparameters

It is well known that training hyperparameters are critical for the model performance of deep neural networks. The batch size represents the number of samples involved in a training iteration. A too-small batch size leads to rough gradient descent and convergence difficulty, while a too-large batch size tends to converge to sharp minimizers and causes poor generalization. A complete traversal of all training samples is called an epoch. A low number of epochs leads to underfitting, and an excessive number of epochs causes overfitting and increases computational costs. The learning rate controls the speed of gradient descent. A larger learning rate is recommended to accelerate the optimization in the early stage of the training process, while a lower learning rate can enable stable convergence to the global optimum later. Therefore, a learning rate decay strategy was adopted to adjust learning rates at different training stages.

In addition to the training hyperparameters, different selections of network structures, activation functions, and optimization algorithms are supposed to affect the model performance of deep neural networks. Therefore, it is necessary to perform comparative studies and determine the optimal settings. Although the exhaustive search among all possible combinations of network structure, training hyperparameters, activation functions, and optimization algorithms is straightforward, the greedy algorithm requires a huge computational cost because the number of combinations of all possible hyperparameters increases exponentially. Therefore, the grid search of four key training hyperparameters (number of hidden layers and neurons inside, learning rate, and regularization coefficient) was performed in this study. Afterward, different activation functions and optimization algorithms were compared as decoupled factors with network structures and training hyperparameters. The regression accuracy between the predicted and actual values of the seismic damage index in the validation dataset was used to evaluate the optimization performance quantitatively.

Figure 5 shows the optimization schematic of the network structures and training hyperparameters through grid search. After the grid search optimization was performed, the following network structure and training hyperparameters were selected, including a hidden layer number of 10, a neuron number of 10 in each hidden layer, a learning rate of 0.25, and a regularization coefficient of 0.016. Figure 6 shows comparisons of model accuracy on validation data using different activation functions and optimization algorithms. The results show that using the sigmoid activation function and Adam optimization algorithm obtained the highest average accuracy and lowest deviancy.

Other training hyperparameters were determined after several trials, including a batch size of 8, an entire training epoch of 1000, a first-order attenuation rate of 0.9, a second-order attenuation rate of 0.999, and a constant preventing zero denominators of 10-8. The established deep neural network for seismic damage index regression was trained using PyTorch 1.8.1 and python3.6 on a 24G GPU of GeForce RTX 3090 manufactured by NVIDIA Corporation from CA, USA.

4. Results and Discussion

4.1. Necessity Verification though Comparative Study between Refined and Original Park-Ang Models

To demonstrate the effectiveness and necessity of the refined Park-Ang model to calculate the seismic damage index from experimental data, this section conducts comparative studies between the refined and original Park-Ang models for different RC columns with various shear span ratios, concrete strengths, and reinforcement ratios, as shown in Figure 7, Figure 8 and Figure 9, respectively. For a more straightforward illustration, the columns are divided into various groups within specific ranges of shear span ratios, concrete strengths, and reinforcement ratios as subfigures. The legend is arranged in a unified manner in the following order: specimen number—parameter type—parameter value—model name.

Figure 7 shows the comparative results of the calculated seismic damage index using the refined and original Park-Ang models with different shear span ratios ranging from 1.5 to 5.5. The elements in the legend are arranged in the order of “Specimen No.—Shear Span Ratio—Refined or Original Park-Ang Model”, where “Park” or “Proposed” represent the original or refined Park-Ang model, respectively. For most RC columns, the original Park-Ang model can only depict a linear damage accumulation process, and the evolution speed in the early stage is even larger than that in the final failure stage (e.g., specimen nos. 18, 20, 27, 23, 51, and 39). As a comparison, the seismic damage index calculated by the refined Park-Ang model increased relatively slowly in the early stage and gradually evolved until the final failure. The results show that the refined Park-Ang model could accurately reflect the nonlinear accumulation of seismic damage under cyclic seismic loads and was superior to the original model considering different shear span ratios of RC columns.

Figure 8 shows the comparative results of the calculated seismic damage index using the refined and original Park-Ang models with different concrete strengths ranging from 20.6 MPa to 118 MPa. Similar phenomena could be observed, including the following: (1) the original Park-Ang model only showed a linear accumulation process of seismic damage with an equal developing speed, and (2) the damage evolution speed of the refined Park-Ang model in the later stage significantly increased showing a trend of exponential growth. The results indicate that the refined Park-Ang model could accurately reflect the nonlinear accumulation of seismic damage (e.g., nos. 11, 51, 85, and 52) and that the performance exceeded the original model for different RC columns with low and high concrete strengths.

Figure 9 shows the comparative results of the calculated seismic damage index using the refined and original Park-Ang models with different reinforcement ratios ranging from 0.0068 to 0.0293. It is further demonstrated that the original model could only reflect a linear damage accumulation process and that the evolution speed of seismic damage during the entire failure process was almost the same. However, the seismic damage index calculated using the refined Park-Ang model increased relatively slowly in the early stage and gradually evolved until the final failure stage. The results further verify that the refined Park-Ang model could accurately reflect the nonlinear accumulation of seismic damage and that it was superior to the original model for RC columns in different reinforcement ratios (e.g., nos. 11, 23, 28, and 67).

The results show that the predicted damage indexes higher than 1 were basically from the original Park-Ang model with dotted lines. The refined Park-Ang model corresponded to the solid lines in these figures, which ranged within an explicit bound of [0, 1] with reasonable physical significance for the damage severity. Therefore, the effectiveness and necessity of the refined Park-Ang model were demonstrated over the original model using different specimens with various shear span ratios, concrete strengths, and reinforcement ratios. The refined Park-Ang model could accurately reflect the nonlinear accumulation process of seismic damage for most RC columns during the entire process of the quasi-static experiments. The seismic damage index of the refined Park-Ang model increased relatively slowly in the early stage and significantly in the later stage. Another advantage of the refined Park-Ang model was verified, i.e., the seismic damage index of the refined model had a unified and explicit bound from 0 to 1 for all RC columns; nevertheless, the upper limit of the seismic damage index from the original model was indeterminate for different specimens.

4.2. Training and Test Results for Seismic Damage Index Regression from Images

Images and quasi-static experimental data of RC columns from nos. 1~100 were utilized for training the established deep neural network, and the remaining data of nos. 101~124 were used to validate the model. As shown in Figure 10a, the training and validation losses decreased during the training process, and the model with the minimum validation loss was chosen as the optimal model to prevent overfitting and improve the prediction ability on other datasets, which is marked as the dashed line in Figure 10a. The prediction error was defined as the scalar difference between the predicted and actual values of the seismic damage index. To further evaluate the prediction error of the seismic damage index quantitatively, the statistical distribution of the prediction error is shown in Figure 10b. It shows that the prediction error of the seismic damage index followed a Gaussian distribution with a mean value around zero and passed the Chi-Square test at a significance level of 95%. The results further demonstrate that the trained deep neural network was unbiased and that the accuracy of the proposed method for the seismic damage index regression of RC columns was verified once again.

Usually, the data-driven model is susceptible to overfitting, which would produce a slight error in the training set and a significant error in the unseen sample in the validation set. Stable without overfitting means that the model performance had slight variations for different samples in the training and validation sets. Unbiased means that the average estimation error between the predicted seismic damage index and the ground truth approximated zero for statistical analysis, indicating that the systematic deviation of the established model was very small and could be omitted.

To predict the seismic damage index from an image of the RC column, the damage-related parameters were extracted in Section 2.3 and used as inputs together with the design-related parameters in order to accomplish the feed-forward process of the well-trained deep neural network (as shown in Figure 3b). Figure 11 shows the corresponding model performances for seismic damage index regression on the training and validation sets. The results indicate that the correlation coefficients between the predicted and actual values of the seismic damage index exceeded 0.98 for both the training and validation sets, suggesting that the model was stable for different samples in the training and validation sets. The validation accuracy was only slightly lower than the training accuracy (the correlation coefficient decreases by 0.2%), indicating that the trained model was not overfitted. Figure 11 shows that 94.3% of predictions fell inside the regression interval with a 95% confidence. The predicted seismic damage index was scattered and uniformly distributed on both sides of the central line, indicating that the proposed method was approximately unbiased with high fitting goodness. Figure 10b also shows the statistical distribution of the prediction error for the seismic damage index. It indicates that the prediction error approximately obeyed a normal distribution, and the mean value was located near zero. The statistical results suggest that the established deep neural network had a good prediction ability for seismic damage index and was stable for different RC columns.

Next, the continuous prediction of the seismic damage index was achieved from a series of images during the entire process of seismic failure. Images of RC columns during the whole period of quasi-static experiments were input to the region-based object detection model of multi-type seismic damage. Bounding boxes for concrete cracks, concrete spalling, and rebar exposure were obtained for each image. Then, the corresponding damage-related parameters were extracted from the recognized bounding boxes and were used as the input variables of the deep neural network together with design-related parameters. Note that the actual seismic damage index was calculated based on the experimental data of quasi-static experiments in Section 2.2.

Data from RC columns nos. 101~124 were validated, and some representative prediction results for the seismic damage index until seismic failure are shown in Figure 12. The results show that the evolution process of seismic damage accumulation was well reflected by the predicted seismic damage index from the input image sequences and design parameters of the RC columns. As for the drop in the predicted seismic damage index, it was actually the effect of randomly selected seismic images. With the increase in the image sequence number (as the seismic damage develops), the predicted seismic damage index of RC columns generally showed a rising trend. However, some drops existed partially. As the viewing angle of the captured seismic images was not fixed and had variations for different specimens, some images of RC columns did not rigorously capture the most severe seismic damage regions, and fluctuations in the predicted seismic damage index existed for these time points. For example, at certain times, one side of the RC column experienced concrete spalling, and another side only experienced a concrete crack. The predicted seismic damage index for the concrete spalling side was larger than that for the concrete crack side, and would inevitably decrease when the viewing point turned from concrete spalling into concrete crack. Figure 13 shows a series of quasi-static experiment images and the corresponding predicted seismic damage indexes for a representative RC column (using no. 123 as an example).

4.3. Robustness Demonstration on New RC Columns with Different Design Parameters

It should be noted that this study established an autonomous regression method for the seismic damage index of RC columns with groups of design parameters. Therefore, it is necessary to further demonstrate the robustness and generalization ability of the proposed method for RC columns with new design parameters for real-world applications.

Table 1. Design parameters of 11 new RC columns for robustness verification.

Specimen No.	Section Size (mm × mm)	Height (mm)	Shear Span Ratio	Axial Compression Ratio	Volumetric Stirrup Ratio (%)
1	225 × 300	1800	5	0.2	1
2	225 × 300	1800	5	0.19	0.5
3	225 × 300	1800	4	0.20	1
4	225 × 300	1800	4.5	0.21	1
5	225 × 300	1800	8	0.11	1
6	225 × 300	1800	10	0.11	0.6
7	400 × 400	1600	4	0.11	0.5
8	400 × 400	1600	7	0.2	0.6
9	400 × 400	1600	3	0.15	0.5
10	350 × 400	2000	8	0.19	0.8
11	350 × 400	2000	4	0.20	0.8

shows the size and design parameters for 11 new RC columns, which were different from the investigated dataset in Section 3.1. The validation dataset in this section was collected from a previous study [61] and from Datacenterhub [62]. A total of 199 images were taken during the quasi-static tests of RC columns. It should be pointed out that the collected images were not continuously taken in the quasi-static test process, and the actual value of the seismic damage indicator for these collected images could not be accurately obtained. Figure 14 shows some representative groups of onsite quasi-static experimental images and the corresponding predicted seismic damage indexes for four different RC columns. The results indicate that the proposed method could reasonably predict the seismic damage index of RC columns with different design parameters under various real-world scenarios from images of quasi-static experiments.

For the input variables of the total concrete crack lengths in horizontal and vertical directions, as well as the areas of concrete spalling and rebar exposure, they were calculated using Equations (5) and (6); the column size was implied using a relative index ratio. This normalization strategy was designed to eliminate the influence of different column parameters because obtaining data samples in the whole parameter space could be impossible. It should be noted that the proposed method cannot be directly applied to obtain the prediction results of the seismic damage index for seismic images of RC columns without prior information on the design parameters. In that case, the required design parameters should be assumed following a particular type of probability distribution, and the seismic damage index is represented by the mean value of a group of predictions with a random sampling of design parameters; this does not belong to the scope of this study and requires future investigations.

4.4. Investigation on the Generalization Ability under Actual Post-Earthquake Scenarios

A typical application scenario for the proposed method for seismic damage assessment is the post-earthquake inspection of RC structures. Images of damaged RC columns can be collected in multiple views from unmanned aerial vehicles, robotics, and cameras. In this section, 133 post-earthquake images of RC columns after Beichuan earthquake M8.0 (Sichuan, China), Meinong earthquake M6.7 (Taiwan, China), Kathmandu earthquake M7.8 (Nepal), and Ecuador earthquake M7.8 were collected. The post-earthquake inspection images and the corresponding damage state annotations have been described in previous studies [63,64].

However, it is difficult to obtain the exact design-related parameters and material properties for these RC components. To address this issue, a fuzzy evaluation method was utilized to verify the feasibility of the proposed method for actual post-earthquake assessment. The damage-related parameters were extracted from inspection images using a region-based multi-type seismic damage recognition method proposed by Xu et al. [36]. As for the design-related parameters, they were assumed to obey the same probability distribution as the training data and were randomly assigned.

For arbitrary column sizes from actual post-earthquake scenarios without actual values, a fuzzy evaluation process was adopted following the assumption that the column size obeyed a statistically pre-acquired probability distribution and was randomly sampled multiple times. It should be noted that this assumption would inevitably bring estimation errors in the seismic damage index regression. However, obtaining an approximate average seismic damage index and classifying an equative seismic damage level from arbitrary inspection images is acceptable for actual post-earthquake scenarios.

Figure 15 shows the representative prediction results of the seismic damage index for RC columns using onsite inspection images after four earthquakes. The results show that the proposed method can basically predict the seismic damage severity of RC columns from actual post-earthquake images in various real-world scenarios.

Because the original experimental data of RC columns in Figure 14 were not released and no experiments were conducted for the actual post-earthquake structures in Figure 15, the actual values of the corresponding seismic damage index could not be calculated. Therefore, only the predicted values are reported in Figure 14 and Figure 15.

Note that because no experiments were conducted for the actual post-earthquake structures in Figure 15, the original experimental data of RC columns could not be obtained, and the actual values of the corresponding seismic damage index could not be calculated. Therefore, the uncertainties were difficult to quantify. The reported method, in which the design-related parameters were assumed to obey the same probability distribution as the training data and were randomly assigned, could be a possible method for conducting the applicable prediction for real-world seismic scenarios. The uncertainty quantification under real-world seismic scenarios without prior design-related parameters would be an interesting issue for further investigation.

4.5. Seismic Damage Level Classification with Unknown Design Parameters

Although the quantitative assessment of the seismic damage index for RC columns can be achieved based on inspection images and quasi-static experimental data, timely classification of seismic damage levels is still a vital issue for rapid post-earthquake assessment. This section investigates an autonomous evaluation method for five seismic damage levels based on the K-means clustering algorithm. Equations of the K-means clustering are omitted to avoid redundancy. The determination criterion of thresholds to separate adjacent seismic damage levels is ensuring that the cumulative probability of seismic damage index inside each level exceeds 95%. Finally, the dividing thresholds for five seismic damage levels, namely no damage (I), minor damage (II), moderate damage (III), severe damage (IV), and destroyed (V), were set as 0~0.015, 0.015~0.21, 0.21~0.42, 0.42~0.8, and 0.8~1.0, respectively. The seismic damage state was defined by inspection experts via onsite assessment based on the technical standard of seismic damage evaluation specification [65]. The seismic damage level classification performance was evaluated based on the confusion matrix compared with the expert scoring of the onsite assessment, including precision and recall for each seismic damage level.

First, 256 images of damaged RC columns from quasi-static experiments in Section 4.1 were investigated, and

Table 2. Confusion matrix of seismic damage level classification for training and validation data.

		Ground Truth by Expert Scoring
Prediction	Seismic Damage Level	I	II	III	IV	V	Precision
	I	16	3	0	0	0	84.2%
	II	3	51	1	0	0	92.7%
	III	0	5	63	1	0	91.3%
	IV	0	0	0	65	2	97.0%
	V	0	0	0	3	43	93.5%
	Recall	84.2%	86.4%	98.4%	94.2%	95.6%

shows the corresponding confusion matrix of the seismic damage level classification. This indicates that the precision and recall for no damage level were the lowest, and the possible reason is that the corresponding images were collected in a relatively small number during the quasi-static experiments. For the damaged levels (I~V), the average precision and recall reached 91.7% and 91.8%, respectively.

Then, 199 images for new RC columns with different design parameters in Section 4.3 were also analyzed, and

Table 3. Confusion matrix of seismic damage levels by images with new design parameters.

		Ground Truth by Expert Scoring
Prediction	Seismic Damage Level	I	II	III	IV	V	Precision
	I	8	6	0	0	0	57.1%
	II	3	27	8	0	0	71.1%
	III	0	4	42	13	0	71.2%
	IV	0	0	3	42	9	77.8%
	V	0	0	0	2	32	94.1%
	Recall	72.7%	73.0%	79.3%	73.7%	78.1%

shows the corresponding confusion matrix of the seismic damage level classification. It also shows that the precision and recall for the no-damage level were the lowest among all the seismic damage levels. The average precision and recall for the damaged levels (II~V) reached 78.6% and 76.0%, respectively. The global accuracy on new RC columns decreased slightly compared with the training and validation dataset.

Furthermore, 133 actual inspection images under four different earthquakes in Section 4.4 were tested, and

Table 4. Confusion matrix of seismic damage levels using images of four actual earthquakes.

		Ground Truth by Expert Scoring
Prediction	Seismic Damage Level	I	II	III	IV	V	Precision
	I	0	0	0	0	0	-
	II	0	0	1	0	0	-
	III	0	0	10	3	0	76.9%
	IV	0	0	2	58	1	95.1%
	V	0	0	0	8	50	86.2%
	Recall	-	-	76.9%	84.1%	98.0%

shows the corresponding confusion matrix of the seismic damage level classification. As the post-earthquake inspection mainly focused on damaged structures, all investigated images were classified into moderate, major, and severe damage levels, with an average precision of 86.1% and an average recall of 86.3%.

The results show that the proposed method could describe the seismic damage level of various images for RC columns under different scenarios of quasi-static experiments and actual earthquakes well. The average classification precision and recall for training and validation data, new quasi-static experimental data, and actual earthquake inspection data reached 82.2% and 82.3%, respectively. It also indicates that the proposed method was stable under different real-world scenarios and gained good robustness and generalization ability for RC columns with various damage-related and design-related parameters.

4.6. Comparison with a Conventional Machine-Learning Method

Comparative studies were performed using the conventional machine-learning-based method to further demonstrate the efficacy and necessity of the proposed computer-vision and machine-learning-based method. A direct convolutional neural network with 10 sequential convolutional blocks was constructed and trained in an end-to-end manner. Each convolutional block contained a convolution layer, a batch normalization layer, a ReLU activation layer, and a max pooling layer, respectively. Each original seismic damage image was used as the input, and the corresponding seismic damage index was utilized as the regression ground truth. The abovementioned convolutional neural network did not embed any seismic knowledge and was regarded as a black-box prediction model from the seismic image to the damage index. The Adam algorithm was utilized for training the convolutional neural network with the same hyperparameters as the proposed method: an initial learning rate of 0.25, a regularization coefficient of 0.016, a batch size of 8, an entire training epoch of 1000, a first-order attenuation rate of 0.9, a second-order attenuation rate of 0.999, and a constant preventing zero denominators of 10-8.

Table 5. Confusion matrix of seismic damage level classification using conventional machine-learning method of convolutional neural network.

		Ground Truth by Expert Scoring
Prediction	Seismic Damage Level	I	II	III	IV	V	Precision
	I	11	4	0	0	0	73.3%
	II	8	51	8	0	0	76.1%
	III	0	4	55	7	0	83.3%
	IV	0	0	1	55	6	88.7%
	V	0	0	0	7	39	84.8%
	Recall	57.9%	86.4%	85.9%	79.7%	86.7%

shows the corresponding confusion matrix for seismic damage level classification. It shows that the precision and recall for the no-damage level were the lowest, and the possible reason for this is that the corresponding images were collected in a relatively small number during the quasi-static experiments. For the damaged levels (I~V), the average precision and recall reached 81.2% and 79.3%, respectively. Compared with the proposed computer-vision and machine-learning-based method in Table 2, the average model performance decreased by 10.5% and 12.5% for precision and recall, respectively. One possible reason for this is that the direct convolutional neural network automatically established the mapping correlation between local regions of seismic damage images and the damage index; however, due to the complicated background and interference objects inside the image, the model could fail to extract the region-of-interest and thus build false connections. The comparison results indicated the efficacy and necessity of the proposed computer-vision and machine-learning-based method against the conventional black box model directly predicting the seismic damage index from images.

5. Conclusions

This study proposes a computer-vision and machine-learning-based framework for seismic damage assessment of RC structures using seismic images and quasi-static experimental data. The main conclusions obtained are as follows:

(1)

A refined Park-Ang model is built to express the coupled effects of structural ductility and energy dissipation and reflects the nonlinear process of seismic damage accumulation. A synthetical indicator of seismic damage with an explicit bound of [0, 1] is designed based on the experimental hysteretic curve. Comparative studies demonstrate the effectiveness and necessity of the refined Park-Ang model for RC components with various shear span ratios, concrete strengths, and reinforcement ratios.

(2)

A deep neural network is established for the damage index regression from multi-type seismic damage and structural design variables. Five damage-related parameters (total horizontal and vertical lengths of concrete cracks, number of concrete cracks, areas of concrete spalling, and rebar exposure) and three design-related parameters (axial compression ratio, shear span ratio, and volumetric stirrup ratio) are utilized as inputs. The optimal network architecture and training hyperparameters are determined using grid search.

(3)

The robustness and generalization ability of machine-learning-based seismic damage index regression are verified using additional RC columns under real-world quasi-static experiments and four actual earthquakes. The results show that the correlation coefficients between the predicted and actual seismic damage index exceed 0.98, and the predicted seismic damage index is unbiased and stable without overfitting.

(4)

An autonomous fuzzy evaluation strategy is proposed for the five-level seismic damage classification of real-world inspection images. The results show that the average classification precision and recall can reach 91.8% and 86.3% for RC seismic images with and without prior information on the material and structural parameters. Compared with the conventional machine-learning-based classification, the average model performance increases by 10.5% and 12.5% for precision and recall, respectively.