Predicting the Compressive Strength of Environmentally Friendly Concrete Using Multiple Machine Learning Algorithms

Abstract

Machine learning (ML) algorithms have been widely used in big data prediction and analysis in terms of their excellent data regression ability. However, the prediction accuracy of different ML algorithms varies between different regression problems and data sets. In order to construct a prediction model with optimal accuracy for fly ash concrete (FAC), ML algorithms such as genetic programming (GP), support vector regression (SVR), random forest (RF), extremely gradient boost (XGBoost), backpropagation artificial neural network (BP-ANN) and adaptive network-based fuzzy inference system (ANFIS) were selected as regression and prediction algorithms in this study; the particle swarm optimization (PSO) algorithm was also used to optimize the structure and hyperparameters of each algorithm. The statistical results show that the performance of the assembled algorithms is better than that of an NN-based algorithm. In addition, PSO can effectively improve the prediction accuracy of the ML algorithms. The comprehensive performance of each model is analyzed using a Taylor diagram, and the PSO-XGBoost model has the best comprehensive performance, with R² and MSE equal to 0.9072 and 11.4546, respectively.

1. Introduction

With the advantages of high strength, low cost and easy construction, concrete has become the most widely used material in the construction of infrastructure and civil buildings. At the same time, the construction of high-difficulty projects such as large-span buildings, docks and cross-sea bridges has led to rising requirements for concrete performance, prompting concrete to develop in the direction of high strength, high flow and high durability. The research shows that fly ash partially replacing cement can effectively improve the durability and mechanical properties of concrete [1,2,3]. In addition, compared with recycled materials such as silica fume, rice husk ash, fly ash and recycled fiber [4] have a wide range of sources, so they have been widely used in the preparation of concrete as a second cementitious material.

It is well known that concrete is a multiphase composite material composed of sand, stone, water, cement and other additives. The complexity of material composition leads to the discrete performance of concrete [5]. In addition, under a variety of physical and chemical actions, the complex mechanism between the above parameters and performance indexes makes the prediction of concrete performance very difficult [6,7]. Therefore, empirical or semi-quantitative methods have been used for a long time to explore the composition and mechanical properties of concrete; when the performance index of concrete reaches a certain value, it is considered that the concrete mixing scheme meets the application requirements, which often leads to the excess performance of concrete, and poor designing efficiency.

With the addition of various additives, it is more and more difficult to accurately predict the performance of concrete using traditional concrete design methods. However, benefiting from the powerful global data analysis and mining capabilities of machine learning algorithms, it is possible to dig deep and discover the inherent laws between input and output parameters regardless of any physical or mechanical model, and then establish a solid and operable accurate relationship [8,9,10]. Based on this, researchers believe that the optimal concrete mixing scheme or structural design can be obtained to meet the performance requirements, thus achieving the purpose of saving construction costs and improving design efficiency [11,12,13,14]. Therefore, there are many advantages to predicting the performance of concrete.

1.1. Literature Review

The performance prediction of concrete has become a research hot spot recently; various ML algorithms have been successfully applied by many studies. Dantas et al. used an ANN algorithm to predict the compressive strength (CS) of concrete by taking the water–cement ratio, fly ash replacement rate and recycled aggregate content as input parameters, and the results showed that ANN had excellent prediction accuracy [15]. Huang et al. combined particle swarm optimization and genetic optimization algorithms with ANN to build a performance prediction model of recycled concrete. The results show that both the hybrid and standalone models have excellent prediction accuracy, but the hybrid model’s is higher [16]. Ahmadi et al. also used a PSO-ANN model to predict the elastic modulus of high-strength and normal concrete, and the conclusion was draw that the proposed model performed excellently [17]. Kim et al. used a genetic algorithm to predict the compressive strength of recycled concrete [18]. Zheng et al. used decision tree, SVR and ANN to construct the performance prediction model of silica fume concrete, and they also used the bagging and boosting methods to assemble the above ML algorithms [19]. After reviewing the recent literature, it was found that the ANN-based modulus is the most wildly used machine learning algorithm for the performance prediction of concrete regarding recycled aggregate concrete [20,21], high-performance concrete [17,22,23], foamed concrete [24,25,26], metakaolin-based concrete materials [27], self-compacted concrete [28], rubberized concrete [29], concrete slabs [30] and other concrete accessories such as steel tubes [31,32], FRP bars [33,34,35,36,37], steel bars [38] and concrete blocks [39].

Although the above machine learning algorithms show a remarkable performance in terms of the specific database, the same ML algorithm may have a poor prediction performance in another database due to the specificity of material composition and regression problems [36]. At the same time, the experimental conditions vary between different research papers, which has a certain negative impact on the accuracy and generalization of ML algorithm prediction [40]. Secondly, there are many factors affecting the performance of concrete, and the regression process of the output and input parameters is very complex, which leads to the fact that an ML algorithm may have excellent prediction accuracy under certain input parameters, while it may be poor under others. Therefore, the model topography and hyperparameters of the existing studies should not be simply applied to specific regression problems. Moreover, the other types of machine learning algorithms are rarely discussed and synthetically compared.

1.2. Objecitves

In view of this, the database of the Gansu provincial transportation research institute Co., Ltd., China (Lanzhou, China), was adopted; 200 data sets were randomly selected, including the mix ratio and CS of FAC in 2023; two ANN algorithms (BP-ANN and ANFIS), two assembled algorithms (RF and XGBoost) and SVR and GP were used to construct the prediction models of the compressive strength of FAC; and a PSO algorithm was used to further optimize the hyperparameters or structures of each model. Finally, the applicability of each model to the constructed database was evaluated using statistical indexes such as R² and MSE. The highlight of the research was the comparison of the prediction performance of various machine learning algorithms and the synthesizing of the PSO algorithm with each predicting model. The objectives of this research were as follows:

• Constructing the predicting model for the compressive strength of concrete containing coal fly ash using six different ML algorithms.
• Synthesizing the standalone models with a PSO algorithm, so as to optimize the hyperparameters of each model automatically.
• Evaluating the applicability of each hybrid ML model using comprehensive statistic indicators.

2. Data Collection

In this study, water (W), cement (C), fly ash (FA), coarse aggregate (A), sand (S) and water-reducing agent (WR) were selected as the influencing factors, which are important components of concrete and have a significant impact on its compressive strength. However, the distribution and accuracy of the data sets greatly affect the prediction accuracy of the algorithm [40,41,42]. Therefore, the collected data were preprocessed as follows:

(1)

The box chart was used to highlight outliers in the data of each input parameter, and then 23 data sets with abnormal distribution were excluded from the 200 data sets; the statistical characteristics of the remaining 177 data sets are shown in

Table 1. Statistical indexes of the data set.

Statistic Index	W (kg/m3)	C (kg/m3)	FA (kg/m3)	A (kg/m3)	S (kg/m3)	WR (kg/m3)	fc (MPa)
Count	177	177	177	177	177	177	177
Mean	157.39	380.71	45.93	1110.27	737.49	5.55	45.54
Std	10.88	69.52	36.66	44.60	61.20	2.37	11.29
Minimum	145.00	189.30	0	999.70	572.90	0	16.30
Maximum	210.00	527.60	129.00	1214.50	920.60	14.10	69.80
Skewness	3.84	−0.12	−0.06	−0.26	−0.06	1.09	−0.14
Mode	153.	442.	0	1136.79	726.80	5.20	36.90
Kurtosis	16.33	−0.42	−1.19	−0.53	−0.53	2.96	−0.61
SEM	0.82	5.23	2.76	3.35	4.60	0.18	0.85

.

(2)

In order to reduce the influence of data scales on the prediction performance and efficiency of the ML algorithm, the data of input parameters were normalized based on Equation (1).

To construct the models with appropriate predicting performance and high generalization, all data sets were randomly divided into a testing set (25%) and training set (75%). The training set was used to fit the features and the target, to endow the models with preliminary predicting performance. The testing set was used to prevent overfitting and ensure the models have excellent generalization.

$$x_i^{\prime }=\frac{x_i-\min (x)}{\max (x)-\min (x)}$$

(1)

where x_i is the data in one input parameter, and max(x) and min(x) are the maximum and minimum value in the corresponding input parameter.

(3)

Subsequently, the database was randomly divided into a training set and testing set through the split function in scikit-learn library, and the proportion of division was 75% for training and 25% for testing.

Figure 1 shows the correlation between all factors (nephogram). Figure 2 shows the correlation between each factor and CS, and the curves on the top and left of the figure show the distribution of the data.

The correlation coefficients between input parameters and CS were calculated, as shown in Figure 3. The correlation between water and water reducer is higher comparing with the others and shows an obvious negative trend; then, the correlation between cement and sand is the second highest. The reason is that sand and coarse aggregate show a negative correlation. Therefore, the more sand, the less coarse aggregate, and the less cement is used in general; the fly ash was added to concrete in a way that partially replaces cement, so the correlation between them is relatively higher.

As for the correlation between the compressive strength and each input parameter, the water, cement and sand are much higher than the others; among them, the influence of water and sand on compressive strength is negative, while the influence of cement on compressive strength is positive. However, the hardening effect of fly ash occurs late [43], so the effect of fly ash on the CS of concrete is small.

3. Machine Learning Algorithms

Machine learning (ML) algorithms have advantages in big data processing, regression and image recognition, and many researchers have applied ML algorithms to scientific research [44,45,46]. In this study, a variety of regression algorithms based on ML technology, including two NN-based algorithms, two assembled algorithms, SVR and GP were used to predict the 7-day CS of concrete containing FA. The accuracy of each model is discussed comprehensively based on various statistical indicators. Figure 4 shows the flowchart of this study.

3.1. PSO Algorithm

PSO is a metaheuristic algorithm. In this algorithm, the potential solution to the problem is to be regarded as a bird (particle) in the flock; each bird has its own speed and position. It is characterized by the ability to update the speed and position of the particle based on the particle’s memory of its own historical optimal solution (extreme value) and the experience shared by the entire population (global extreme value); the characteristic is formulaic, as shown in Equations (2) and (3).

$$v_{iD}^{\prime }=\omega \times v_{iD}+c_1{r}_1\left(p_{iD}-x_{iD}\right)+c_2{r}_2\left(p_{gD}-x_{iD}\right)$$

(2)

$$x_{iD}^{\prime }=x_{iD}+v_{iD}^{\prime }$$

(3)

where the subscript D indicates that there is a D-dimensional search space that is the number of indicators to be searched; $v_{iD}$ represents the velocity of the i particle in the D; $x_{iD}$ represents the position of the particle; $p_{iD}$ and $p_{gD}$ represent the individual extremum and global extremum, respectively; $\omega$ is the inertia weight, and the value usually takes 1; $c_1$ and $c_2$ are the acceleration constant, which usually take $c_1=c_2=2$; and $r_1$ and $r_2$ are the random numbers generated from the interval [0, 1].

The hyperparameters of an ML algorithm have a significant impact on its prediction accuracy, while manual adjustment is cumbersome and time-consuming, thus significantly reducing the efficiency of an ML algorithm. Therefore, in this study, a PSO optimization algorithm is used to automatically find the best value of hyperparameters that have an important influence in each ML algorithm.

However, ANFIS requires huge random access memory on the part of the computer in the operation process, so it is not suitable for PSO to optimize its hyperparameters across a wide range. Moreover, the most influential hyperparameters of ANFIS are the type and number of membership functions and membership grade, so the hyperparameter optimization of ANFIS used trial and error. In addition, the hyperparameters of the others standalone ML algorithms adopted their default values or the literature recommendations. The specific hyperparameters of each hybrid and standalone ML model are shown in

Table 2. Hyperparameters setting of the standalone models.

ML Model	Value of Hyperparameters
BPNN	Two hidden layers, and the first layer has 18 neurons, the second layer has 12 neurons.
RF	n_estimate = 15, random state = 45, max_depth = 3
SVR	kernel = rbf
XGBoost	default
GP	population_size = 5000, generations = 20, stopping_criteria = 0.01, p_crossover = 0.7, p_subtree_mutation = 0.1, p_hoist_mutation = 0.1, p_point_mutation = 0.1, max_samples = 0.9, verbose = 1, parsimony_coefficient = 0.01, random_state = 0
ANFIS	membership type = graussf, membership grade = (2, 2, 2, 2, 2, 2, 2, 2)

and

Table 3. Searched hyperparameters of each predicting model.

PSO Hyperparameters Setting	Predicting Model	Searching Hyperparameters
population size = 20 generation = 20	BPNN	Neurons number of each hidden layer
	RF	n_estimators, random state, max_depth
	XGBoost	max_depth, learning_rate, n_estimators
	SVR	C, epsilon, gamma
	GP	population_size, generations stopping_criteria, max_samples, verbose, parsimony_coefficient, random_state

.

3.2. BP-ANN

BP-ANN is a computational system built by imitating the operation mode of neurons [47]. It is usually composed of an input layer, hidden layer and output layer, and its operation mode is mainly composed of forward propagation and backpropagation.

In forward propagation, the data of each input parameter are first input into the neurons of the “input layer”, and then the input data are transmitted to the next neuron through a certain logical relationship, as shown in Equation (4), and finally to the neurons of the “output layer”.

$$y_j=f\left(v_j\right)=f\left({\displaystyle \sum _{i=1}^n{w}_{ij}{x}_i+b_j}\right)$$

(4)

This logical relationship includes the weight w_ij of the I neuron of the k layer to the j neuron of the k + 1 layer (k is the number of the layers, including the input layer and the hidden layers), and the bias b_i of the k layer neuron. The received data of the k + 1 layer will be processed by the activation function; then, the data will be transformed to the k + 2 layer.

In backpropagation, the predicted value y, obtained through the aforementioned logical relationship, is compared with the experimental value y, and the loss function is obtained, so the gradient function of the loss function can be obtained. Finally, the weight and bias are constantly transformed through the gradient descent algorithm, and the optimal weight vector and the bias of each layer are finally obtained.

3.3. ANFIS

This method combines a neural network with fuzzy logic to classify the data. This fuzzy classification ability enables the computer to understand the input data, and then it has a higher ability to resist data error and excellent prediction performance [36].

ANFIS models are usually divided into five layers:

(1)

The first layer is the fuzzy layer, which undertakes fuzzy processing of input data by membership function. The selection of the type and number of membership function is usually subjective. When there are more membership functions of each input parameter, there will also be more membership degrees, so more if–then rules will be generated, which may improve the prediction accuracy to a certain level but also significantly increase the requirement of computer performance.

(2)

The second layer is to calculate the firing strength of each if–then rule.

(3)

The third layer normalizes the firing strength and obtains the trigger intensity of the if–then rule relative to the others.

(4)

The fourth layer calculates the output value of each if–then rule by multiplying the original input data and the relative trigger intensity obtained in the third layer.

(5)

The fifth layer is the output layer, which weights and sums the output values obtained in the fourth layer and defuzzies them.

Since the final output result is weighted and summed, it also means that the predicted target can only be a single variable, and it is impossible to achieve a multi-variable output.

3.4. SVR

The support vector machine (SVM) is proposed by Vapnik et al. [48]; the algorithm used for regression is support vector regression (SVR), whose main idea is to construct an interval band that can accommodate as many data points as possible, while minimizing the loss of data points that are not in the interval band.

Data points in the interval band are not counted for their loss, and SVR maps the data to the feature space by kernel function, which makes the nonlinear regression problem become a linear regression problem approximately. Therefore, the advantages of SVR include high generalization, being iterative fast, global optimization and avoidance of local minimization [19,49]. In addition, SVR has an excellent fitting performance in nonlinear regression problems with multiple variables and small data.

3.5. XGBoost

XGBoost is an algorithm based on a boosting framework. Boosting is an addition model that includes multiple estimators, each of which gives a set of predicted values, and the latter estimator will learn the deviation between the previous estimator’s predicted value and experimental value, and then continuously reduces the deviation.

3.6. RF

Random forest is an assembled machine learning algorithm based on a bagging framework, which combines multiple decision trees. Bagging is to randomly extract n samples from the database and form a new training set. According to the above method, M new training sets are generated according to the assembled M decision trees; each decision tree gives a set of prediction values base on its training set, and then M prediction results are obtained. Finally, the average of these M prediction results is used as the final prediction result.

3.7. GP

Genetic programming is an evolutionary algorithm designed to fit data sets by giving a mathematical expression that approximates the relationship between features and target. Unlike others regression ML algorithms, GP can construct a mathematical expression by searching and combining basic mathematical operators, features and constants. Therefore, GP is also one of the few interpretable ML algorithms, that is, one giving exact mathematical expressions.

4. Results and Discussions

4.1. Prediction Performance of Standalone Models

R², MSE, MAE and explanatory variance were used to evaluate the prediction accuracy of the constructed algorithms. The statistic indexes were calculated using the r2_score, mean_absolute_error, mean_squared_error and sm.taylor functions in the sklearn.metrics library.

The prediction results of each standalone model are shown in Figure 5, Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10. MSE represents the degree of error between the predicted and experimental value. At the same time, the weight of large error will be highlighted after the squared, so it is more sensitive to the error than other statistical indexes. Although the prediction error of the testing set was slightly higher than that of the training set, the standalone model still had acceptable MSE values except GP and ANFIS.

The R² indicates the fitness of prediction of the model. As can be seen from the figure, the R² of the predicted and experimental values is generally around 0.8, indicating the selected ML algorithms have a good predicting performance. However, the R² of AFNIS and GP are only 0.7015 and 0.7154, respectively. The prediction ability of the GP model is dependent on variability, just like genetic variation [26]. Therefore, the variation direction of branches is highly controlled by the hyperparameter settings. In contrast, SVR has the highest R² by virtue of its excellent generalization. As for RF and XGBoost models, their fitting goodness showed a certain fluctuation, and this is because of their random splitting of tree branches and the formation of data subsets of each sub-tree integrated in them, and all of this results in a decisive dependency on the hyperparameter setting. As a contrast, the SVR model will map the lower dimension problem to the higher one and simplify the calculation through kernel function, performing little randomness. Therefore, the SVR model is outperformed. The ANFIS showed a worse R²; this may because the database has six features, resulting in the requirement of many more membership degrees (MDs), and the MDs only default at two for each feature, resulting in the poor performance of the ANFIS model.

4.2. Prediction Performance of Hybrid Models

The prediction value and performance of the hybrid models are shown in Figure 11, Figure 12, Figure 13, Figure 14, Figure 15 and Figure 16. After PSO optimization, the prediction accuracy of each model has been significantly improved. MSE and R² of PSO-RF are 11.9172 and 0.9035, respectively, which are 42.9% lower and 8.7% higher than that of RF, respectively; while for PSO-SVR, the MSE and R² are reduced 14.8% and increased 1.9% compared with its standalone model; for XGBoost, the results are 44.1% and 8.9%; for GP, they are 56.2% and 22.4%; and for BP-ANN, they are 16% and 2.9%.

Among them, the GP model improved the most after PSO optimization, which indicates that hyperparameters have a greater impact on the GP model to a certain extent. At the same time, PSO has the smallest improvement on the SVR model, which is because the SVR model has a high generalization, so the impact of data quality and model setting on its prediction accuracy is relatively low. The PSO-XGBoost model has the lowest error and the highest fitness, with an MSE of 11.4594 and R² of 0.9072. In addition, the assembled and SVR algorithms have better prediction performance than the NN-based algorithms.

The optimization of the ANFIS model used trial and error due to its high requirements on computer performance. Under the condition that there are no more than three membership grades for each input parameter, the membership grade combination of 2-3-2-3-1 with Gaussian function was finally determined. After the optimization, the MSE was reduced by 34.3%, and the R² was increased by 18.4%. As aforementioned, the predicting performance of RF and XGBoost is highly dependent on the hyperparameter setting. Therefore, the predicting accuracy is significantly increased.

4.3. Analysis of Error Distribution

The absolute error distribution of each model is shown in Figure 17, Figure 18, Figure 19, Figure 20, Figure 21 and Figure 22. After PSO optimization, the maximum absolute error of each model is reduced to some extent. However, the distribution of red noise in the RF, XGBoost, ANFIS and BP-ANN models is relatively scattered, indicating that there are many large error points in the range of predicted data. In contrast, the error distribution of SVR and PSO-GP is more uniform in the predicted data range, so there are fewer large error points.

Although RF and XGBoost have a higher R² and lower MSE after PSO optimization, they have more large error points, which may lead to larger prediction errors in practical applications. Therefore, SVR and PSO-GP are more suitable for regression prediction of this data set.

It is worth noting that the prediction accuracy of the GP model is relatively lower, but the absolute error distribution is more uniform, indicating that there are fewer relatively large error points in its prediction results, which indicates that the GP model has good prediction potential for this data set.

4.4. Accuracy Analysis

The statistical indicators of the predicted and experimental values of each model are summarized in

Table 4. Statistic indexes of each model.

ML Model	Evaluating Index
	R2	MSE	STD	MAE
RF	0.8309	20.885	8.8893	3.6563
PSO-RF	0.9035	11.9172	9.8887	2.6271
SVR	0.8872	13.9315	10.7003	2.7257
PSO-SVR	0.9038	11.8761	10.5086	2.5996
XGBoost	0.8340	20.5032	10.3703	3.4999
PSO-XGBoost	0.9072	11.4594	10.6130	2.3637
GP	0.7154	35.1627	11.0829	4.6226
PSO-GP	0.8753	15.4052	10.5057	2.9886
BP-ANN	0.8368	20.1649	11.5004	3.6589
PSO-BP-ANN	0.8630	16.9292	10.7190	3.2411
Optimized ANFIS	0.8303	26.4208	12.3607	3.3869
ANFIS	0.7015	40.2328	11.0996	4.1215

. The R² of the optimized ANFIS is only 4.5% and 0.7% lower than XGBoost and RF, respectively, but the MSE is 29.6% and 27.7% higher. In addition, the change rates of R² and MSE of each model are significantly different after PSO optimization. The reason is that different statistic indicators have different sensitivity to the errors. Therefore, it is difficult to fully reflect the prediction accuracy of the model through a single statistical indicator.

In view of this, a Taylor diagram was used to combine several statistical indicators to comprehensively evaluate the prediction accuracy of each model. According to Equation (5), the R, RMSE and standard deviation (STD) of the predicted value and experimental value are combined in the two-dimensional plane graph. The comprehensive evaluation of each model is shown in Figure 23.

The blue dot–dash line represents the R, the green dash line represents the RMSE and the black dotted line represents the standard deviation. The distance between each point to Ref represents the comprehensive performance of each ML model; a shorter distance means a higher comprehensive performance. As can be seen from the figure, PSO-XBGoost has the best comprehensive performance.

$$RMS{E}^2=ST{D}^2{}_{exprimental}+ST{D}^2{}_{predicted}-2ST{D}^2{}_{exprimental}ST{D}^2{}_{predicted}R$$

(5)

5. Conclusions

To build a compressive strength prediction model for fly ash concrete, this study selected two assembled algorithms, two NN-based algorithms and machine learning algorithms. The PSO algorithm was also used to optimize the structure or hyperparameters of the above ML algorithms. Finally, the predicting performance of each model was evaluated using statistical indicators. The main conclusions of this study are as follows:

(1)

As a standalone model, the SVR algorithm has the highest R² of 0.8837 and lowest MSE of 13.9315 with good generalization. In addition, the assembled algorithm outperforms the NN-based algorithm.

(2)

The PSO algorithm can effectively improve the prediction accuracy of all the ML models. Among them, the improvement in prediction accuracy of GP is the highest; its MSE decreased by 56.2% and R² increased by 22.4% after cooperating with PSO. In addition, the R² of the PSO-RF, PSO-XGBoost and PSO-SVR models are all greater than 0.9.

(3)

The absolute error distribution of the PSO-GP and SVR algorithms is relatively uniform, which means that there are fewer large error points in their prediction results, so it is not easy to have a large prediction error under a certain set of features. According to the statistical indicators of each standalone and hybrid algorithm, PSO-XGBoost has the best comprehensive performance.

(4)

Given the specificity of each predicting scenario, the same predicting models which have an appropriate accuracy in the f_c prediction may not have performed excellently in the other scenarios such as anti-chloride diffusion, carbonization and so forth. Therefore, the applicability of each model should be carefully discussed in the others’ predicting scenarios.

(5)

Although six different machine learning algorithms were used to predict the f_c of the concrete containing coal fly ash, the kinds of machine learning algorithms are still limited. Future research could discuss the applicability of other machine learning algorithms, even constructing a synthesizing operational interface to improve usability in the field.