4.1. Experimental Setting
In this study, the dataset, including 270 samples and nine predictor variables, is used to train and test the proposed machine learning approach for estimating the CS of RCC containing recycled aggregates. The proposed method, named GBO-XGBoost, is an integration of a metaheuristic algorithm and an advanced gradient boosting machine. The gathered dataset was separated into a training set and a testing set. The ratio of training-to-testing samples is set to be 9/1 to ensure sufficient samples for training a robust regressor. Moreover, to negate the effect of randomness in data sampling, the model training and testing phases have been repeated 20 times. The final model’s performance is determined by the average outcomes obtained from these multiple runs. In this study, the XGBoost regressor is constructed with the assistance of the Python library provided in [
81]. The GBO search engine is implemented with the MEALPY library [
82]. The experiments with the constructed model are performed on the Dell G15 5511 (Core i7-11800H and 16 GB RAM).
Notably, the variables in the current dataset have different ranges. For instance, the minimum and maximum values of the cement quantity are 142 and 375, respectively. Meanwhile, the quantity of recycled coarse aggregate varies from 403.05 to 1264.40. Therefore, data normalization is required to pre-process the data. This step can help avoid circumstances in which the variables with large magnitudes dominate the ones with small magnitudes. In this study, the Z-score method is used because it can help normalize the mean and standard deviation measures of the variables [
83]. The Z-score normalization equation is expressed as follows:
where
and
refer to the normalized and original variables, respectively.
and
represent the mean and standard deviation of the original variable, respectively.
It is also noted that performance measurement is crucial for assessing the predictive capability of machine learning models. In the current work, various approaches are used to evaluate the goodness-of-fit achieved by different predictors. The employment of different metrics helps observe the predictive capability of a model from different angles. For regression analysis, the root mean square error (RMSE), mean absolute percentage error (MAPE), and coefficient of determination (
R2) are commonly employed. The equations used to compute these metrics are given by:
where
ti and
yi refer to the actual and predicted CS values of the
ith data sample, respectively.
N denotes the total number of data samples.
RMSE and MAPE are the two indices providing measurements regarding the average prediction error committed by the model. Meanwhile, the coefficient of determination indicates the proportion of variation in the target variable explained by the model. In addition, analyzing the characteristics of the model’s residuals is also essential in model evaluation. The model’s residual (∆) is computed as
, where
t and
y are the measured and estimated CS values of one concrete mix. Assessing the range of the ∆ may help shed light on how well the data is fitted by the model. Therefore, this study relies on various techniques for investigating the feature of the model’s residual, including range inspection, regression error characteristic curve [
84], and statistical descriptions.
To demonstrate the advantages of the newly developed GBO-XGBoost, ANN, SVM, and M5-MT are used as benchmark approaches. The reason is that ANN and SVM have been shown to be prominent approaches for predicting the CS of recycled aggregate concrete [
21]. In addition, good performance of M5-MT for CS estimation has been reported in various studies [
77,
85]. In this study, the ANN and SVM are constructed in MATLAB’s statistics and machine learning toolbox [
86]. The M5-MT is built via the functions of the library provided in [
87]. As pointed out in [
21],
k-fold cross-validation is the most common approach for machine learning-based predictions of the mechanical properties of recycled aggregate concrete. This approach helps ensure the robustness and generalizability of the developed models. Hence, the cross-validation processes [
88] are used to perform the model selection phases and identify suitable hyper-parameters of the benchmark approaches.
As mentioned earlier, the current work resorts to GBO for optimizing the performance of the XGBoost model. The flowchart of the proposed framework is presented in
Figure 4. Given the training and testing datasets, the structure of XGBoost is initiated. In each iteration, the GBO evaluates the quality of the candidate solutions, consisting of the XGBoost’s hyper-parameters. These hyperparameters are the number of individual trees, the learning rate, the maximum tree depth, the
L2-regularization factor, and the
L1-regularization factor. The lower and upper boundaries of the searched hyper-parameters are [1, 0.001, 1, 0, 0] and [200, 1, 10, 1000, 1000], respectively.
Herein, the model selection process of XGBoost is formulated as a global optimization problem in which a cost function is minimized. In detail, the following objective function is designed for the GBO searching process:
where APE refers to the average prediction error of the model;
NTree denotes the number of individual trees in the XGBoost ensemble; MTD represents the maximum tree depth;
and
are the
L2-regularization factor and
L1-regularization factor, respectively;
wTN = 0.001,
wMTD = 0.01, and
wReg = 0.0001 are the weighting coefficients of the terms in the objective function.
The average prediction error (APE) mentioned above is the mean of the testing results obtained from a five-fold cross-validation process. APE is computed as follows:
where
K = 5 is the number of data folds; RSME refers to the root mean square error.
It can be observed that the aforementioned objective function is composed of a cross-fold evaluation and the terms used for expressing the model’s complexity. Notably, to minimize fGBO, the metaheuristic algorithm needs to locate candidate solutions featuring low APE. However, reducing the prediction error often entails an increasing degree of the model’s complexity. In this study, to restrict the overcomplex model’s structure, the terms governing the complexity of XGBoost are taken into account. It is noted that the GBO searching process does not only favor the model having good predictive accuracy (i.e., a low value of APE), but also pays attention to reducing the number of NTree and MTD. The reason is that the lower the NTree and MTD are, the less sophisticated the model’s structure is. Moreover, high values of regularization coefficients (i.e., and ) impose high pressure on restricting the complexity of the XGBoost model so that it is less susceptible to overfitting. Relying on fGBO, the GBO metaheuristic carried out the searching process with 20 population members and during 100 iterations. When the optimization is accomplished, the performance of the best model is evaluated using the metrics of RMSE, MAPE, and R2.
4.2. Prediction Results and Performance Comparison
As mentioned in the previous section, the proposed approach relies on GBO to optimize the performance of XGBoost. The predictive capability of XGBoost depends on the setting of its hyper-parameters: the number of individual trees (
NTree), the learning rate (L
R), the maximum tree depth (MTD), the
L2-regularization factor (
), and the
L1-regularization factor (
). After 100 iterations, the population-based metaheuristic helped identify the following configuration of the XGBoost model used for estimating the CS of recycle aggregate RCC:
NTree = 46, L
R = 1.97, MTD = 4,
= 168.96, and
= 0.00011. The optimization progress of GBO is illustrated in
Figure 5.
In addition, based on the cross-validation process [
88], the ANN trained by the LM algorithm is configured as follows: the number of neurons in the hidden layers is 11 and the learning rate of the algorithm is 0.001. The SVM model necessitates the setting of the penalty coefficient (
CSVM), the radial basis kernel function parameter (
), and the margin of tolerance (
). These hyperparameters of the SVM model are set as follows:
CSVM = 100,
= 1, and
= 0.1. In the case of M5-MT, the minimum leaf size and the splitting thresholding value are set to be 2 and 10
−6, respectively.
The experimental results of the proposed GBO-XGBoost and the benchmark methods are summarized in
Table 4. The CS estimation outcomes of the proposed model and other approaches are graphically demonstrated by the scatter plots in
Figure 6. Observably, the data points predicted by GBO-XGBoost are clustered around the agreement line. Meanwhile, there is a large degree of dispersion of the data points in the scatter plots yielded by the benchmark models. This fact indicates that the accuracy of the data fitting provided by GBO-XGBoost is higher than that yielded by other models. Quantitatively speaking, the proposed method has attained an outstanding performance with a RMSE of 2.639, a MAPE of 7.823%, and an R
2 of 0.941. This outcome is better than SVM (RMSE = 4.048, a MAPE of 12.027%, and an R
2 of 0.866) as the second-best method by a large margin. ANN (RMSE = 4.493, a MAPE of 12.912%, and an R
2 of 0.835) and M5-MT (RMSE = 5.855, a MAPE of 17.692%, and a R
2 of 0.722) attained performance that is significantly inferior to that of GBO-XGBoost.
RMSE is deemed the most commonly employed metric for assessing the quality of a regressor. This metric is essentially the average deviation between a model’s estimated values and the measured values of CS. It can be shown that the typical differences between the actual and predicted results obtained from GBO-XGBoost, SVM, ANN, and M5-MT are 2.639, 4.048, 4.493, and 5.855 MPa, respectively. Since deviations or residuals represent the distance between the constructed functional mapping and the actual data samples, RSME typically exhibits the degree of dispersion of the model’s residuals. In other words, this index provides insight on how closely the actual CS values cluster around the constructed functional mapping. RSME entails the squaring of residuals; therefore, this index is sensitive to samples associated with exceptionally large residuals.
Notably, the RSME of SVM is roughly 1.5 times larger than the result of GBO-XGBoost. This fact indicates that the prediction outcome of the proposed method contains fewer exceptionally large residuals than the benchmark approach. Moreover, if the distribution of residuals is assumed to follow a normal distribution, it can be roughly estimated that 95% of the prediction results of GBO-XGBoost lie within the range of ±2 × RMSE, or approximately ±5.3 MPa. Following the aforementioned assumption, the deviation range of the benchmark approaches is at least ±8.1 MPa, which is significantly wider than that of the proposed method.
In addition, MAPE also provides an intuitive interpretation of a model’s performance in terms of relative deviation. This index basically computes the average magnitude of the residuals. As observed from the results, the average relative deviation between the actual and predicted CS yielded by GBO-XGBoost is about 7.8%. Based on the classification of MAPE values for CS estimation stated in [
32], a MAPE < 10% indicates excellent predictive capability of the newly developed method. The results of the benchmark methods falling in the range of [10–20%] imply their good performance. These classification criteria based on MAPE are understandable because the task of predicting CS values is highly challenging. The reason is that the prediction error is inevitable due to the variations in the physical features of the concrete mixes’ constituents as well as the uncertainty in the processes of casting and testing the concrete specimens [
89].
The R
2 index also provides crucial information about the goodness of fit of a machine learning model. This coefficient quantifies the proportion of the variation in the CS that is predictable from its influencing factors and the trained model. This index ranges from 0 to 1. An R
2 of 1 exhibits a perfect data-fitting result. With an R
2 of 0.94, the proposed GBO-XGBoost is capable of explaining up to 94% of the variations in the CS of recycle aggregate RCC. The R
2 values obtained from other benchmark models are all less than 90%. For instance, SVM, as the second-best approach, is able to predict about 87% of the variations in the target variable.
Figure 7 graphically shows the improvements in the performance measurement metrics gained by the proposed model. In detail, compared to SVM as the best benchmark approach, GBO-XGBoost helps reduce the RSME and MAPE indices by roughly 35%. The proposed approach also enhanced the goodness of fit in terms of R
2 by at least 8.72%.
In
Figure 8, detailed distributions of the models’ relative residual range (
r) are graphically demonstrated. In the task of CS prediction, it is desired to attain values of
r that are smaller than 20% [
32]. Moreover, the information about how a model’s residuals are allocated within the range of [0%, 20%] is valuable to assess its prediction performance [
10]. Accordingly, the current work investigated the proportion of
r in five groups: (i)
r ≤ 5%, (ii) 5 <
r ≤ 10%, (iii) 10% <
r ≤ 15%, (iv) 15% <
r ≤ 20%, and (v)
r > 20%. As observed from the figure, GBO-XGBoost attained the highest proportion of data samples in the first group. The ratios of data in this group obtained from the proposed method, SVM, ANN, and M5-MT are 48.89%, 31.48%, 34.26%, and 23.70%, respectively. Consider the relative range of ≤10%; GBO-XGBoost excels other models with the proportion of roughly 73%. SVM, ANN, and M5-MT attained outcomes of roughly 60%, 56%, and 43%, respectively. Notably, only about 9% of the data samples predicted by GBO-XGBoost are associated with
r exceeding 20%. In this regard, the performance of the newly developed method is superior to other models by a large margin. The proportions of
r exceeding 20% yielded by SVM, ANN, and M5-MT are approximately 16%, 20%, and 30%, respectively.
Moreover, to assess the model’s performance with respect to the cumulative distribution of the residuals, this study relies on the regression error characteristic curve (REC) proposed in [
84]. A graph of REC is constructed by sorting the absolute value of the residuals. The analysis results are provided in
Figure 9. Herein, the
x-axis is the absolute value of prediction error. The
y-axis displays the proportion of the magnitude of the residual that is less than a certain value in the
x-axis. For example, a point in a REC with coordinates (6.00, 0.90) means that 90% of the absolute value of the residuals is less than 6 MPa. Due to the characteristics of the REC, the more data points in the upper left corner of the graph, the better the prediction performance. Therefore, the accuracy of the prediction result can be judged by how fast the REC reaches the line of
y = 1. Based on such observations, the area under the REC (AUC) can be calculated to express the model’s prediction performance. XGBoost-GBO attained the highest AUC of 0.90; this outcome is followed by SVM (AUC = 0.85), ANN (AUC = 0.84), and M5-MT (0.78).
Additionally, the distributions of the prediction performance obtained from the machine learning models are provided in
Figure 10. Herein, the RMSE index is selected to measure the prediction errors committed by the models. Each box plot in the figure displays the minimum, first quartile, median, third quartile, and maximum of RSME values obtained from 20 experiments. It is apparent that the median of the predictor error obtained from GBO-XGBoost (2.61) is significantly lower than that yielded by other models. The medians of RMSE attained by SVM, ANN, and M5-MT are 3.65, 4.65, and 5.71, respectively. Therefore, the median prediction error estimated by GBO-XGBoost is at least 28% lower than that of other benchmark methods. To reliably affirm the superiority of the proposed method, this study resorts to the Wilcoxon signed-rank test [
90] with a threshold of
p-value = 0.05. Pairwise hypothesis tests (GBO-XGBoost vs. SVM, GBO-XGBoost vs. ANN, and GBO-XGBoost vs. M5-MT) all yielded
p-values of 0.0001 < 0.05. Therefore, it is confident to reject the null hypothesis of indifferent prediction performances and confirm the pre-eminence of the proposed method in the task of estimating the CS of recycled aggregate RCC.
Moreover, to assess the effect of potential outliers in the dataset, this study utilizes the criterion based on using the standard deviation. Herein, the value of 2 times the standard deviation is employed as the threshold for outlier filtering [
91]. This approach is based on the observation that, for a normal distribution, approximately 95.4% of the data points are likely to fall within two standard deviations from the mean. Based on the analysis with the CS values, it is found that 10 samples in the dataset have the target output surpassing ±2×σ
CS, where σ
CS denotes the standard deviation of the CS values. To appraise the influence of these 10 samples on the GBO-XGBoost, this study has performed experiments consisting of two scenarios: (i) Scenario 1: The original dataset consisting of 270 samples; and (ii) Scenario 2: The data set without 10 samples identified as potential outliers. The experimental outcomes are reported in
Table 5. As can be seen from the table, the model trained with 260 samples in Scenario 2 suffers from a slight reduction in prediction accuracy. Its RMSE and MAPE have been increased to 2.742 and 8.382%, respectively. Moreover, the model trained with fewer data samples can only explain 92% of the variability in the target output, a minor deterioration compared to 94% in the original model (Scenario 1). Thus, it can be concluded that the samples having CS values exceeding ±2×σ
CS are relevant for the model construction and should be included in the analysis with GBO-XGBoost.
4.4. Reduction of Overestimation Based on GBO-XGBoost Using Asymmetric Loss Function
This section first analyzes the characteristics of the residuals obtained from GBO-XGBoost trained by the SEL function. As mentioned in the previous section of the paper, SEL is used to guide the model’s training phase to minimize the overall deviation between the actual and predicted output. This loss function does not take into account the sign of the residuals. It is reminding that the residual (Δ) is calculated as
t −
y, where
t and
y refer to the actual and estimated CS values, respectively. In this regard, a positive Δ implies an underestimation of the CS value. On the contrary, an overestimation of the CS value is equivalent to a negative Δ. Since the standard SEL cannot express the preference for underestimation of the target variable, it is expected that the proportions of negative and positive residuals yielded by GBO-XGBoost are relatively close to 50%. In addition, the average value of the residuals should be close to 0. The distribution of the residuals yielded by the model using the standard SEL is demonstrated by the histogram in
Figure 12. As expected, the ratio of negative to positive Δ is close to 50:50. In addition, the model’s residuals cluster around the mean of −0.01. The dispersion of the distribution is characterized by a standard deviation of 2.67. The skewness value of −0.12 indicates that the left tail of the distribution is slightly longer than its right one.
To improve the safety and reliability of the prediction results, this study resorts to the ASEL. This asymmetric loss function introduces an additional parameter (γ) to the standard SEL for expressing the preference for underestimation. Notably, γ should be set to be ≥1; and the bigger the value of γ, the stronger the preference against negative residuals. To implement ASEL in XGBoost, it is required to modify the first-order (
) and second-order (
) derivatives of the loss function as follows:
where γ is the tuning parameter of the ASEL;
t and
y denote the measured and estimated CS of the RCC, respectively.
To optimize the XGBoost model using ASEL, this study proposes the inclusion of the preference for underestimation of the target variable in the objective function of the GBO metaheuristic. In this regard, the revised objective function of GBO is given by:
where APPR represents the average proportion of positive residuals; other terms in the equations are similar to those previously defined in
(refer to Equation (12)).
The objective function stated in Equation (16) helps exhibit the three goals of constructing the XGBoost model for CS estimation: minimizing the prediction error, maximizing the generalization property, and the preference for positive residuals. With each setting of γ (the tuning parameter of the ASEL), the GBO is used to optimize the XGBoost model. Through experiments, it was found that γ ≥ 5 brought about major improvements in terms of APPR. A large value of this tuning parameter tends to increase the value of APPR but also deteriorates the overall prediction performance. Notably, γ = 15 helps yield the most desired outcome, with an average proportion of positive residuals of 66.30% and a coefficient of determination of 0.929. The value of γ > 15 could not bring about any improvement in APPR. The experimental results regarding the effect of different values of γ are illustrated in
Figure 13.
After 100 iterations, the GBO has identified the following configuration of the XGBoost using ASEL: the number of individual decision trees = 51, the learning rate = 0.70, the maximum depth of the tree = 4, the L
2-regularization coefficient = 275.07, and the L
1-regularization coefficient = 0.029. It is noted that this model configuration is associated with γ = 15. The characteristic of the residuals obtained from the GBO-XGBoost using ASEL is demonstrated by the histogram in
Figure 14. As shown in this figure, the proportion of overestimation has been drastically reduced from 50.74% to 33.70%. The whole distribution is shifted towards the right side of the
x-axis. The residuals cluster around the mean of 0.97, and the standard deviation of the distribution is 2.84. A positive skewness of 0.13 indicates that the right tail of the distribution has been broadened. Notably, the distribution has shifted to the right side of the
x-axis to accommodate more positive residuals.
The overall prediction performance of GBO-XGBoost using ASEL is demonstrated by the scatter plot in
Figure 15. With RMSE = 2.89, MAPE = 7.87%, and R
2 = 0.93, there is excellent agreement between the actual and estimated results. In addition, the majority of the data points (66.3%) have been observed to be below the line of best fit. With MAPE < 10%, the prediction performance of GBO-XGBoost using ASEL is highly desired and surpasses that of other benchmark methods. The newly developed method is capable of explaining 93% of the variation in the CS variable.
A comparison between the ASEL- and SEL-based models is graphically presented in
Figure 16. As shown in this figure, the ASEL-based model is capable of reducing the number of overestimations by roughly 17%; meanwhile, the asymmetric loss function only causes a minor deterioration in the overall prediction performance. The MAPE value of the GBO-XGBoost using ASEL is only 0.05% less than that of the model using the standard SEL. Therefore, the newly developed model, which combines the advantages of GBO and XGBoost, can be a potential alternative for estimating the CS of RCC using recycled aggregates. To facilitate the implementation of GBO-XGBoost, a graphical user interface (GUI) based on the proposed framework was developed. The GUI is constructed with the help of the tkinter library [
95]. The developed GUI is demonstrated in
Figure 17 and has been deposited in the Github repository at
https://github.com/NHDDTUEDU/RA_RCC_GBO_XGBoost (accessed on 22 July 2024). Herein, the program requires input information about the RCC’ constituents and the curing age. The user can also select the option of using either the SEL- or ASEL-based models via the radio buttons provided in the GUI.