Plum Tree Algorithm and Weighted Aggregated Ensembles for Energy Efficiency Estimation

Abstract

This article introduces a novel nature-inspired algorithm called the Plum Tree Algorithm (PTA), which has the biology of the plum trees as its main source of inspiration. The PTA was tested and validated using 24 benchmark objective functions, and it was further applied and compared to the following selection of representative state-of-the-art, nature-inspired algorithms: the Chicken Swarm Optimization (CSO) algorithm, the Particle Swarm Optimization (PSO) algorithm, the Grey Wolf Optimizer (GWO), the Cuckoo Search (CS) algorithm, the Crow Search Algorithm (CSA), and the Horse Optimization Algorithm (HOA). The results obtained with the PTA are comparable to the results obtained by using the other nature-inspired optimization algorithms. The PTA returned the best overall results for the 24 objective functions tested. This article presents the application of the PTA for weight optimization for an ensemble of four machine learning regressors, namely, the Random Forest Regressor (RFR), the Gradient Boosting Regressor (GBR), the AdaBoost Regressor (AdaBoost), and the Extra Trees Regressor (ETR), which are used for the prediction of the heating load and cooling load requirements of buildings, using the Energy Efficiency Dataset from UCI Machine Learning as experimental support. The PTA optimized ensemble-returned results such as those returned by the ensembles optimized with the GWO, the CS, and the CSA.

1. Introduction

The nature-inspired optimization research field evolved a lot in recent years, and its sources of inspiration are various, ranging from swarm intelligence [1] to the laws of physics [2]. Nature-inspired optimization algorithms are often used as an alternative to mathematical methods to solve global optimization problems in various domains, such as finance [3], engineering [4], data mining [5], and other areas. Their main advantage is that they lead to good optimal solutions using reasonable computational resources. Several representative classes that can be distinguished among nature-inspired algorithms [6] are swarm intelligence methods [7], evolutionary computation approaches [8], and the algorithms which have the laws of physics and biology as their main source of inspiration [9].

Methods in the swarm intelligence class represent solutions as swarm members that exchange information during the search process. The exchanged information is related to the properties of the swarm leaders, which are either global leaders or local leaders. This class includes algorithms such as the Particle Swarm Optimization (PSO) algorithm [10], the Elephant Herding Optimization (EHO) algorithm [11], the Whale Optimization Algorithm (WOA) [12], the Spotted Hyena Optimizer (SHO) [13], and the Horse Herd Optimization Algorithm (HHOA) [14].

Approaches in the evolutionary computation class associate the search process to the evolution of a set of solutions called the population. Representative algorithms of this class are the Genetic Algorithms (GA) [15], the Differential Evolution (DE) algorithm [16], the Scatter Search (SS) algorithm [17], and the Invasive Weed Optimization (IWO) algorithm [18].

The algorithms inspired by the laws of biology and physics include a selection of algorithms which imitate the biological and physical processes of the nature. This class is represented by algorithms such as Harmony Search (HS) [19], the Gravitational Search Algorithm (GSA) [20], the Fireworks Algorithms (FA) [21], and the Spiral Dynamics Algorithm (SDA) [22].

Table 1. Representative Nature-Inspired Algorithms for 2005–2022.

Year	Nature-Inspired Algorithm	Source of Inspiration
2005	Artificial Bee Colony (ABC) [23]	honeybee foraging behavior
2006	Cat Swarm Optimization (CSO) [24]	seeking and tracing mode behaviors
2007	Monkey Search (MS) [25]	behavior of monkeys looking for food
2008	Fast Bacterial Swarming Algorithm (FBSA) [26]	foraging mechanism of E-coli bacterium
2009	Cuckoo Search (CS) [27]	brood parasitic and Levy flight behaviors
2010	Bat Algorithm (BA) [28]	echolocation behavior
2011	Cuckoo Optimization Algorithm (COA) [29]	egg laying and breeding behaviors
2012	Flower Pollination Algorithm (FPA) [30]	flower pollination process
2013	Dolphin Echolocation (DE) [31]	echolocation ability of dolphins
2014	Forest Optimization Algorithm (FOA) [32]	tree survival durations
2015	Runner-Root Algorithm (RRA) [33]	plants propagating through runners
2016	Crow Search Algorithm (CSA) [34]	crows’ intelligent behavior
2017	Salp Swarm Algorithm (SSA) [35]	salp fish swarming
2018	Tree Growth Algorithm (TGA) [36]	trees’ competition for light and foods
2019	Sailfish Optimizer (SO) [37]	sailfish group hunting
2020	Horse Optimization Algorithm (HOA) [38]	horse herds’ hierarchical organization
2021	African Vultures Optimization Algorithm (AVOA) [39]	African vultures’ behavior
2022	Snake Optimizer (SO) [40]	snake mating behavior

presents a selection of representative nature-inspired algorithms for each year from the period 2005–2022.

As shown in the table, the sources of inspiration are various, ranging from the behaviors of specific birds or mammals to processes related to flower pollination or to trees’ competition for light. Moreover, the big number of nature-inspired algorithms proposed in recent years demonstrates that the nature-inspired optimization research field is still an open research field with a variety of challenges.

The applications of nature-inspired algorithms are various. The authors of [41] considered an algorithm based on the Glowworm Swarm Optimization (GSO) algorithm [42] and the Bacterial Foraging Optimization (BFO) algorithm [43] for multi-objective optimization. The approach presented in [44] introduced a novel algorithm named the Pelican Optimization Algorithm (POA), which was applied in engineering problems such as the pressure vessel design problem and the welded beam design problem. Another approach, the one presented in [45], considered a binary version of the HOA with applications in feature selection for classification problems. Nature-inspired algorithms were also considered in research fields such as image processing [46], pattern recognition [47], data mining [48], and video compression [49].

The application of these machine learning models in real systems is crucial to achieve smart building control and performant energy efficiency, as demonstrated in [50], in which the authors considered current practical issues related to the application of machine learning models to building energy efficiency. Moreover, the application of ensemble learning received a lot of attention in building energy prediction recently [51] because it returned predictions which were more stable and accurate than the ones returned by using conventional methods based on a single predictor. Using the building energy prediction data from an institutional building located at the University of Florida as experimental support, the authors of [51] demonstrated the feasibility of a method based on an exhaustive search when identifying the optimal ensemble learning model. The building environment sector is responsible for approximately one-third of the world’s energy consumption; therefore, seeking solutions that aim to reduce the demands of the building energy is an important research direction in the context of the mitigation of adverse environmental impacts [52].

In addition to the application of machine learning techniques to improve energy efficiency estimation, the research community also considered approaches to improve energy storage devices. In [53], the authors proposed a solution based on a mechanically interlocked network (MIN) in a Li anode, which promises future success in battery fields that suffer from volume variations. As shown in [54], the lithium metal batteries which use solid electrolytes are expected to become the next generation of lithium batteries. Another challenge related to lithium-ion batteries is the accurate prediction of their lifetime in the context of the acceleration of their technological applications and advancements [55].

The main objective of this paper is the proposal of a novel nature-inspired algorithm with applications in a wide range of engineering problems. The energy efficiency estimation problem was selected as a representative use-case for the application of the proposed algorithm because of the importance of energy research in the context of climate change’s near-future impact as well as because the application of ensemble learning to building energy efficiency prediction attracted much attention in the research community in recent years.

The main contributions of the article are:

(1)

A critical review of the nature-inspired approaches applied to machine learning ensemble optimization;

(2)

An overview of the application of machine learning approaches to energy efficiency estimation;

(3)

The proposal of a novel nature-inspired algorithm called the Plum Tree Algorithm (PTA), which has the biology of the plum tree as its main source of inspiration;

(4)

The evaluation and the validation of the PTA using 24 benchmark functions and the comparison of the results to the results obtained by using the Chicken Swarm Optimization (CSO) algorithm [56], the PSO algorithm, the Grey Wolf Optimizer (GWO) [57], the CS algorithm, the CSA, and the HOA;

(5)

The application of the PTA to weight optimization for an ensemble of four algorithms for energy efficiency data prediction using the Energy Efficiency Dataset from the UCI Machine Learning repository [58,59];

(6)

The comparison of the results obtained using the PTA ensemble approach to the ones based on other nature-inspired algorithms and to the results obtained in other research studies.

The article has the following structure. Section 2 presents the research background. Section 3 presents the Plum Tree Algorithm (PTA), including its sources of inspiration and its main steps. Section 4 presents the results obtained after the comparison of the PTA to six nature-inspired algorithms using 24 benchmark objective functions. Section 5 presents the application of the PTA to weight optimization for an ensemble of four regressors used for energy efficiency estimation. Section 6 presents a discussion and compares the results to the ones obtained using other methods. Finally, Section 7 presents conclusions and future research directions.

2. Background

The research background section is organized into three subsections. The first subsection reviews the application of nature-inspired algorithms to the optimization of machine learning ensembles, the second section presents representative studies which considered the application of machine learning techniques to energy efficiency estimation, and the third section presents the contribution of this paper with respect to previous research.

2.1. Nature-Inspired Algorithms for Machine Learning Ensemble Optimization

The authors of [60] considered the application of ensemble and nature-inspired machine learning algorithms for driver behavior prediction. The three classifiers applied were the Support Vector Machine (SVM) algorithm, the K-Nearest Neighbors (K-NN) algorithm, and the Naïve Bayes (NB) algorithm. Initially, the performance of the classifiers was improved using techniques such as voting ensemble, bagging, and boosting. Then, four nature-inspired algorithms were applied to improve the results. These four algorithms were as follows: the PSO algorithm, the GWO, the WOA, and the Ant Lion Optimizer (ALO) [61]. The results showed that the best approach was the GWO-voting approach, which had an accuracy of 97.50%.

The approach presented in [62] analyzed driver performance considering a learning technique based on weighted ensembles and nature-inspired algorithms. The applied ensemble technique was the Extreme Learning Machine (ELM) algorithm, and the combined classifiers were the Generalized Linear Model (GLM), the K-NN algorithm, and the Linear Discriminant Analysis (LDA) algorithm. The applied nature-inspired algorithms were the GA, the Grasshopper Optimization Algorithm (GOA) [63], and the binary BA. Compared to the other models proposed in that approach, the model that was based on the hybrid nature-inspired ELM algorithm led to the highest performance metrics.

In [64], the authors applied an ensemble method based on the SVM, the K-NN, and the PSO to improve the accuracy of intrusion detection. The obtained results suggested that the novel approach was better than the Weighted Majority Algorithm (WMA) [65] in terms of accuracy. The approach presented in [66] applied a WOA-based ensemble using the SVM and the K-NN for the classification of the diabetes based on data collected from medical centers from Iran. The WOA ensemble classifier improved the best preceding classifier by about 5%.

The authors of [67] proposed a method based on an improved version of the PSO combined with the Adaptive Boosting (AdaBoost) [68] ensemble algorithm for the classification of imbalanced data. The experimental results showed that the proposed method was effective in the processing of the data characterized by a high imbalance rate. Another approach, the one considered in [69], proposed an ensemble system based on the modified AdaBoost with area under the curve (M-AdaBoost-A) classifier, which was optimized using strategies such as the PSO. The proposed approach returned performant results both for 802.11 wireless and for traditional enterprise intrusion detection.

2.2. Machine Learning Approaches to Energy Efficiency Estimation

Energy efficiency estimation using machine learning techniques was considered in the literature from different perspectives. Moreover, the literature contains a variety of review studies which focus on the application of machine learning techniques to energy performance, such as the ones presented in [70,71].

In [72], the authors proposed an estimation model for heating energy demand in which a methodology based on a two-layer approach was developed using a database of approximately 90,000 Energy Performance Certificates (EPCs) of flats from Italy’s Piedmont region. The proposed methodology consisted of two layers: the classification layer, which was used to estimate the segment of energy demand, and the regression layer, which was used to estimate the Primary Energy Demand (PED) of the flat. The four algorithms which were used and compared in both layers were the Decision Tree (DT) algorithm, the Support Vector Machine (SVM) algorithm, the Random Forest (RF) algorithm, and an Artificial Neural Network (ANN). Another approach, the one presented in [73], considered the application of two ANNs, one for actual energy performance and another for key economic indicators. The data was collected from the energy audits of 151 public buildings from four regions of South Italy. The prediction of their energy performance was done using a decision support tool based on these two ANNs.

In [74], an ensemble learning technique method was applied to energy demand prediction in the case of the residential buildings. The data was collected from residential buildings located in Henan, China. The applied model was used to forecast the heating load 2 h ahead. The ensemble learning method combined the algorithms Extreme Learning Machine (ELM), Extreme Gradient Boosting (XGB), and Multiple Linear Regression (MLR) with the Support Vector Regression (SVR) algorithm to obtain a model that was more accurate. The authors of [75] considered a PSO-based ensemble learning approach for the energy forecast. The proposed optimized ensemble model was used for the management of the smart home energy consumption. The PSO algorithm was used to fine-tune the hyper-parameters of the ensemble model. The results showed that the performance of the optimized ensemble was better than the performance of both the non-optimized ensemble and the individual models.

The authors of [76] developed an ensemble machine learning model for the prediction of building cooling loads. The model was created and evaluated using data from 243 buildings. The results show that cooling loads can be predicted quickly and accurately in the early design stage. Energy demand management was approached in [77] using a method based on a weighted aggregated ensemble model. The ensemble consisted of the weighted linear aggregation of the Least Square Boosted Regression Trees (LSB) and Gaussian Process Regression (GPR) algorithms, and the design parameters were evaluated using the Marine Predators Algorithm (MPA) [78].

Building energy prediction was approached in [79] using energy consumption pattern classification. The DT algorithm was used for mining energy consumption patterns and classifying energy consumption data, whereas ensemble learning was used to establish energy consumption prediction models for the patterns. The data used in the work was hourly meteorological data collected from a meteorological station, and the energy consumption data was collected from an office building from New York City.

2.3. Contributions with Respect to Previous Research

Compared to the method presented in [80], where ensembles of ANN were used for the prediction of smart grid stability, the ensemble proposed in this manuscript consists of four regressors. The same objective function for the evaluation of the performance of the ensembles was considered, but the results were compared to more nature-inspired algorithm-based ensembles, using 30 runs for each fold.

The authors of [81] approached the problem of energy efficiency estimation using the same experimental dataset as the one used in this manuscript. However, they focused on heating predictions only rather than on both cooling and heating predictions. Even though they used nature-inspired algorithms such as the Firefly Algorithm (FA) [82], the Shuffled Complex Evolution (SCE) algorithm [83], the Optics-Inspired Optimization (OIO) algorithm [84], and the Teaching–Learning-Based Optimization (TLBO) algorithm [85], those algorithms were used to tune the hyper-parameters of an ANN rather than to tune ensembles of regressors such as in the approach presented in this article. The conclusions of the authors showed that the prediction results were performant and better than the results obtained by using state-of-the-art algorithms.

The approach presented in [86] applied the Shuffled Frog Leaping Algorithm (SFLA) [87] in the tuning of the hyper-parameters of a Regression Tree Ensemble (RTE), which was an approach proposed for the accurate prediction of heating loads and cooling loads. Compared to other methods presented in the article, the SFLA-based approach showed the best results in terms of evaluation metrics.

In [88], the PSO algorithm was applied to improve the performance of the Extreme Gradient Boosting Machine (XGBoost) algorithm [89] used in the prediction of heating loads. The PSO-based approach returned better results compared to classical state-of-the-art approaches. However, the authors of the study focused only on the prediction of the heating load like the approach in [81]. Similar to the approach presented in this article, 80% of the dataset was considered in the training phase, and 20% was considered in the testing phase. On the other hand, that approach also considered a re-sampling method based on 10-fold cross-validation to reduce errors.

3. Plum Tree Algorithm

The plum is a fruit of the Prunus species, a genus of trees which also includes peach, cherry, nectarine, almond, and apricot trees. Plums have a lot of health benefits. They contain a lot of nutrients, are rich in antioxidants, promote bone health, and help lower blood sugar. Several statistics mention China and Romania as the top two largest plum producers.

Figure 1 shows a representative illustration of plum flowers.

Figure 2 shows a representative illustration of a bunch of plums.

The main sources of inspiration for the PTA are as follows:

• The flowering of the plum trees in the early spring;
• The transformation of the pollinated flowers to plums;
• The plums drop before maturity by 20–30% due to various reasons or diseases, such as diseases caused by plant viruses [90];
• The plums have a shelf life of around 2–6 weeks, even when they are stored at 0 °C [91].

The PTA implements these sources of inspiration as follows: the positions of the flowers and the positions of the plums are represented as matrices, the dropping of the plums before maturity is represented by a fruitiness threshold ($FT$), and the shelf life of the plums after they are collected is represented by a ripeness threshold ($RT$). The two thresholds lead to the development of three types of equations to update the positions of the flowers.

The PTA presents similarities with other bio-inspired algorithms which were also sources of inspiration. The PTA was inspired by the CSO algorithm in the use of a Gaussian distribution to update the positions of the flowers and the use of a random number from a range bounded by the minimum fruitiness rate ($F{R}_{min}$) and the maximum fruitiness rate ($F{R}_{max}$). The PTA was inspired by the PSO algorithm in the development of the equations that update the positions of the flowers. Another source of inspiration is the GWO in the sense that these equations consider the best and second-best positions determined so far. The PTA describes these positions as the ripe position and the unripe position, respectively. The CSA inspired the PTA in the use of an additional data structure for the flowers, the updating of the plums in each iteration considering the best values between the current positions of the plums and the new positions of the flowers, and the use of a random numerical value in the range of $\left[0, 1\right]$ to distinguish between the three types of equations to update the positions of the flowers.

Figure 3 presents the high-level overview of the PTA.

The PTA starts with the initialization of $N$ flowers and $N$ plums in the $D$-dimensional search space (Step 1). Initially, the positions of the plums have the same values as the positions of the flowers. The fitness values of the flowers and the plums are computed in Step 2, and the $p_{gbest}$ is initialized to the position of the plum which has the best fitness value (Step 3). The current iteration $iter$ has the value 1 initially (Step 4). The instructions from Step 5–Step 11 are performed $I$ times, where $I$ is the number of iterations. The positions of the ripe and the unripe plums correspond to the positions of the plums with the best and the second-best fitness values, respectively (Step 5). According to the value of a random number r in the range of $\left[0,1\right]$ (Step 6), the positions of the flowers are updated according to the fruitiness phase, the ripeness phase, or the storeness phase equations using Formulas (3), (4), and (5) and (6), respectively (Step 7). The three phases are delimited by their $FT$ and $RT$ thresholds. The flowers’ positions are adjusted to be within the limits of the search space (Step 8), and the positions of the plums are updated using Formula (7) (Step 9). The $p_{gbest}$ is updated if there exists a plum which has a better fitness value (Step 10), the current iteration $iter$ is incremented by 1 (Step 11), and the algorithm returns the $p_{gbest}$ value after all $I$ iterations have been performed (Step 12).

The pseudo-code of the PTA is presented below (see Algorithm 1).

Algorithm 1 PTA
1:	Input $I, D, N, FT, RT, F{R}_{min}, F{R}_{max}, ϵ, OF, X_{min}, X_{max}$
2:	Output $p_{gbest}$
3:	initialize $N$ flowers in the $D$-dimensional space with values from $\left[X_{min}, X_{max}\right]$
4:	initialize $N$ plums to the positions of the $N$ flowers
5:	apply $OF$ to compute the fitness of the plums and the flowers and update $p_{gbest}$
6:	for $iter=1$to $I$ do
7:	compute the ripe position $P_{ripe}$
8:	compute the unripe position $P_{unripe}$
9:	for each flower do
10:	update $r$ to a random number from the range [0, 1]
11:	if $r \ge FT$ then
12:	update the flower using formula (3)
13:	else if $r \ge RT$ then
14:	update the flower using formula (4)
15:	else
16:	update the flower using the formulas (5)–(6)
17:	end if
18:	adjust the flowers to be in the range $\left[X_{min}, X_{max}\right]$
19:	end for
20:	for each plum do
21:	update the plum using formula (7)
22:	end for
23:	update the $p_{gbest}$
24:	end for
25:	return $p_{gbest}$

The input of the PTA is represented by the following parameters: $I$—the number of iterations, $D$—the number of dimensions, $N$—the number of plums, $FT$—the fruitiness threshold, $RT$—the ripeness threshold, $F{R}_{min}$—the minimum fruitiness rate, $F{R}_{max}$—the maximum fruitiness rate, $ϵ$—a constant for avoiding division by zero, $OF$—the objective function, $X_{min}$—the minimum possible value of the position, and $X_{max}$—the maximum possible value of the position. The output of the PTA is $p_{gbest}$, the global best plum position.

$N$ flowers are initialized in (line 3) in the $D$-dimensional search space with values from the range $\left[X_{min},X_{max}\right]$:

$$flowers=\left[\begin{array}{ccc}{F}_{1,1}^0& \dots & F_{1,D}^0\\ \dots & \dots & \dots \\ F_{N,1}^0& \dots & F_{N,D}^0\end{array}\right],$$

(1)

and $N$ plums are initialized in (line 4) with the values of the flowers:

$$plums=\left[\begin{array}{ccc}{P}_{1,1}^0& \dots & P_{1,D}^0\\ \dots & \dots & \dots \\ P_{N,1}^0& \dots & P_{N,D}^0\end{array}\right]=flowers.$$

(2)

The fitness values of the flowers and plums are computed in (line 5) using the $OF$, and the value of the global best plum $p_{gbest}$ is updated to the position of the plum which has the best fitness value.

The instructions from the (lines 7–23) are performed $I$ times, such that $iter$ describes the value of the current iteration.

The ripe position $P_{ripe}$ is updated to the position of the plum with the best fitness value (line 7), whereas the unripe position $P_{unripe}$ is updated to the position of the plum with the second-best fitness value (line 8).

The positions of the flowers are given as $F_i^{iter}$, where $i=\overline{1, N}$ are updated in (lines 10–18) considering the value of a random number $r$ within the range of $\left[0, 1\right]$ (line 10).

If the value of $r$ is greater than or equal to $FT$ (line 11), then the position of the flower is updated using the following formula:

$$F_i^{iter}=F_i^{iter-1}+random\left(F{R}_{min},F{R}_{max}\right)\times \left(P_i^{iter-1}-F_i^{iter-1}\right),$$

(3)

such that $random\left(F{R}_{min},F{R}_{max}\right)$ returns uniformly a number from the range $\left[F{R}_{min},F{R}_{max}\right]$.

If the value of $r$ is less than $FT$ and greater than or equal to $RT$ (line 13), then the position of the flower is updated as follows:

$$F_i^{iter}=F_i^{iter-1}+2\times r_1\times \left(P_{ripe}-F_i^{iter-1}\right)+2\times r_2\times \left(P_{unripe}-F_i^{iter-1}\right),$$

(4)

where $r_1$ and $r_2$ are random numbers within the range of $\left[0, 1\right]$, and $P_{ripe}$ and $P_{unripe}$ are the ripe position and the unripe position, respectively.

If the value of $r$ is less than $RT$ (line 15), then the flower updates its position using the formula:

$$F_i^{iter}=P_i^{iter-1}\times \left(1+\mathcal{N}\left(0,{\sigma }^2\right)\right),$$

(5)

such that $\mathcal{N}\left(0,{\sigma }^2\right)$ is a Gaussian distribution which has the mean 0 and the standard deviation σ², which is defined by the formula:

$${\sigma }^2=\{\begin{array}{c}\hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}1, \hspace{1em}\mathrm{if} OF\left(P_i^{iter-1}\right)<OF\left(P_{ripe}\right),\\ e^{\frac{OF\left(P_{ripe}\right)-OF\left(P_i^{iter-1}\right)}{\left|OF\left(P_i^{iter-1}\right)\right|+ϵ}}, \mathrm{otherwise},\hfill \end{array}$$

(6)

where $ϵ$ is a constant used to avoid division by $0$.

The flowers are adjusted to be in the range of $\left[X_{min},X_{max}\right]$ (line 18) such that if $F_{i,j}^{iter}<X_{min}$, then $F_{i,j}^{iter}=X_{min}$, and if $F_{i,j}^{iter}>X_{max}$, then $F_{i,j}^{iter}=X_{max}$, where $j=\overline{1,D}$.

The position of each plum is updated in (lines 20–22) as follows:

$$P_i^{iter}=\{\begin{array}{c}{F}_i^{iter},\hspace{1em} \mathrm{if} OF\left(F_i^{iter}\right)<OF\left(P_i^{iter-1}\right),\\ P_i^{iter-1}, \mathrm{otherwise}.\hfill \end{array}$$

(7)

The value of the global best plum $p_{gbest}$ is updated in (line 23) to the position of the plum which has the best fitness value according to the objective function OF. Finally, the algorithm returns the $p_{gbest}$ value (line 25).

4. Results

The PTA was evaluated and compared to the CSO algorithm, the PSO algorithm, the GWO, the CS algorithm, the CSA, and the HOA using 24 objective functions. Out of the 24 objective functions, 23 were from the EvoloPy library [92,93,94], whereas one was from [95]. The implementations of the PSO algorithm, the GWO, and the CS algorithm were the ones from the EvoloPy library, whereas the implementations of the CSO algorithm, the CSA, and the HOA were developed in-house.

The CSO algorithm version used in experiments was an adapted version in which the equations for the updating of the positions of the hens were modified such that the values of the $S_1$ and $S_2$ parameters were set to 1 in cases in which the fitness of the hen was better than the fitness of the associated rooster or hen, respectively. The mother hens were considered to be the top MP (mothers percent) hens.

The experiments were written in Python using the EvoloPy library and a machine with the following properties: Intel(R) Core(TM) i7-7500U CPU processor, 8.00 GB installed RAM, 64-bit operating system, and Windows 10 Pro N operating system.

Table 2. Unimodal Objective Functions (Objective Functions $f_1-f_7$).

Objective Function	Mathematical Formula	Range	N	Optimal Value
$f_1$	${{\displaystyle \sum }}_{i=1}^N{x}_i^2$	[−100, 100]	30	0
$f_2$	${{\displaystyle \sum }}_{i=1}^N\left|x_i\right|+{{\displaystyle \prod }}_{i=1}^N\left|x_i\right|$	[−10, 10]	30	0
$f_3$	${{\displaystyle \sum }}_{i=1}^N{\left({{\displaystyle \sum }}_{j=1}^i{x}_i\right)}^2$	[−100, 100]	30	0
$f_4$	max (|xi|, 1 ≤ i ≤ N)	[−100, 100]	30	0
$f_5$	${{\displaystyle \sum }}_{i=1}^{N-1}\left[100\times {\left(x_{i+1}-x_i^2\right)}^2+{\left(x_i-1\right)}^2\right]$	[−30, 30]	30	0
$f_6$	${{\displaystyle \sum }}_{i=1}^N{\left(\left[x_i+0.5\right]\right)}^2$	[−100, 100]	30	0
$f_7$	${{\displaystyle \sum }}_{i=1}^{N}i\times x_i^4+r,$where $r$ is a random number from $\left[0,1\right)$	[−1.28, 1.28]	30	0

,

Table 3. High-Dimensional Multi-Objective Functions (Objective Functions $g_1-g_7$).

Objective Function	Mathematical Formula	Range	N	Optimal Value
$g_1$	${{\displaystyle \sum }}_{i=1}^N-x_i\sin \left(\sqrt{\left|x_i\right|}\right)$	[−500, 500]	30	−12,569.487
$g_2$	${{\displaystyle \sum }}_{i=1}^N\left[x_i^2-10\times \cos \left(2\pi x_i\right)+10\right]$	[−5.12, 5.12]	30	0
$g_3$	$-20\times exp\left(-0.2\sqrt{\frac{1}{N}{{\displaystyle \sum }}_{i=1}^N{x}_i^2}\right)-exp\left(\frac{1}{N}{{\displaystyle \sum }}_{i=1}^N\cos \left(2\pi x_i\right)\right)+20+e$	[−32, 32]	30	0
$g_4$	$\frac{1}{4000}\times {{\displaystyle \sum }}_{i=1}^N{x}_i^2-{{\displaystyle \prod }}_{i=1}^N\cos \left(\frac{x_i}{\sqrt{i}}\right)+1$	[−600, 600]	30	0
$g_5$	$\frac{1}{2000}\times {{\displaystyle \sum }}_{i=1}^N{x}_i^2-{{\displaystyle \prod }}_{i=1}^N{\text{cos}}^4\left(\frac{x_i^2}{i}\right)+1$	[−300, 300]	30	0
$g_6$	$$\begin{gathered} \frac{\pi }{N}\times \left\{10\times si{n}^2\left(\pi y_1\right)+{{\displaystyle \sum }}_{i=1}^{N-1}{\left(y_i-1\right)}^2\times \left[1+10\times si{n}^2\left(\pi y_{i+1}\right)\right]+{\left(y_N-1\right)}^2\right\}+ {{\displaystyle \sum }}_{i=1}^{N}U\left(x_i,10,100,4\right), \\ \mathrm{where}{y}_i=1+\frac{x_i+1}{4}, U\left(x_i,a,k,n\right)=\{\begin{array}{cc}k{\left(x_i-a\right)}^n,& x_i>a;\\ 0,& -a\le x_i\le a\\ k{\left(-x_i-a\right)}^n,& x_i<-a,\end{array}; \end{gathered}$$	[−50, 50]	30	0
$g_7$	$$\begin{gathered} 0.1\times \left\{si{n}^2\left(3\pi x_1\right)+{{\displaystyle \sum }}_{i=1}^N{\left(x_i-1\right)}^2\times \left[1+si{n}^2\left(3\pi x_i+1\right)\right]+{\left(x_N-1\right)}^2\times \left[1+si{n}^2\left(2\pi x_N\right)\right]\right\}+ {{\displaystyle \sum }}_{i=1}^{N}U\left(x_i,5,100,4\right), \\ \mathrm{where} U\left(x_i,a,k,n\right)=\{\begin{array}{cc}k{\left(x_i-a\right)}^n,& x_i>a;\\ 0,& -a\le x_i\le a\\ k{\left(-x_i-a\right)}^n,& x_i<-a.\end{array}; \end{gathered}$$	[−50, 50]	30	0

and

Table 4. Fixed-Dimensional Multimodal Objective Functions (Objective Functions $h_1-h_{10}$).

Objective Function	Mathematical Formula	Range	N	Optimal Value
$h_1$	${\left(\frac{1}{500}+{{\displaystyle \sum }}_{j=1}^{25}\frac{1}{j+{{\displaystyle \sum }}_{i=1}^2{\left(x_i-a_{ij}\right)}^6}\right)}^{-1}$	$\left[-65.356, 65.356\right]$	2	0.998
$h_2$	${{\displaystyle \sum }}_{i=1}^{11}{\left[b_i-\frac{x_1\left(c_i^2+c_i{x}_2\right)}{c_i^2+c_i{x}_3+x_4}\right]}^2$	$\left[-5, 5\right]$	4	0.00030
$h_3$	$4x_1^2-2.1x_1^4+\frac{1}{3}{x}_1^6+x_1{x}_2-4x_2^2+4x_2^4$	[−5, 5]	2	−1.0316
$h_4$	${\left(x_2-\frac{5.1}{4{\pi }^2}{x}_1^2+\frac{5}{\pi }{x}_1-6\right)}^2+10\left(1-\frac{1}{8\pi }\right)\cos x_1+10$	$\left[-5, 5\right]$	2	0.3978
$h_5$	$\left[1+{\left(x_1+x_2+1\right)}^2\left(19-14x_1+3x_1^2-14x_2+6x_1{x}_2+3x_2^2\right)\right]$$\times \left[30+{\left(2x_1-3x_2\right)}^2\times \left(18-32x_1+12x_1^2+48x_2-36x_1{x}_2+27x_2^2\right)\right]$	$\left[-2, 2\right]$	2	2.999
$h_6$	$-{{\displaystyle \sum }}_{i=1}^4{c}_{i}exp \left(-{{\displaystyle \sum }}_{j=1}^3{a}_{ij}{\left(x_j-p_{ij}\right)}^2\right)$	[0, 1]	3	−3.86
$h_7$	$-{{\displaystyle \sum }}_{i=1}^4{c}_{i}exp \left(-{{\displaystyle \sum }}_{j=1}^6{a}_{ij}{\left(x_j-p_{ij}\right)}^2\right)$	[0, 1]	6	−3.32
$h_8$	$-{{\displaystyle \sum }}_{i=1}^5{\left[\left(X-a_i\right){\left(X-a_i\right)}^T+c_i\right]}^{-1}$	[0, 10]	4	−10.1532
$h_9$	$-{{\displaystyle \sum }}_{i=1}^7{\left[\left(X-a_i\right){\left(X-a_i\right)}^T+c_i\right]}^{-1}$	[0, 10]	4	−10.4029
$h_{10}$	$-{{\displaystyle \sum }}_{i=1}^{10}{\left[\left(X-a_i\right){\left(X-a_i\right)}^T+c_i\right]}^{-1}$	[0, 10]	4	−10.5364

present the unimodal objective functions, the high-dimensional multi-objective functions, and the fixed-dimensional multimodal objective functions, respectively.

For each algorithm and objective function, 30 runs were performed. The number of iterations was set to 1000, and the population size was set to 50 for all runs.

Table 5. Nature-Inspired Algorithms’ Specific Configurations.

Nature-Inspired Algorithm	Specific Configurations
	Parameter	Value
CSO	hierarchical order update frequency ($G$)	10
	minimum flight length ($F{L}_{min}$)	0.5
	maximum flight length ($F{L}_{max}$)	0.9
	zero division avoiding constant ($ϵ$)	1.0 × 10300
	roosters percent ($RP$)	20
	hens percent ()	60
	mother hens percent ($MP$)	10
PSO	minimum velocity ($V_{min}$)	−6
	maximum velocity ($V_{max}$)	6
	minimum inertia ($w_{min}$)	0.2
	maximum inertia ($w_{max}$)	0.9
	cognitive component constant ($c_1$)	2
	social component constant ($c_2$)	2
GWO	parameter $a$	2
CS	discovery rate of alien eggs ($p_a$)	0.25
CSA	flight length ($fl$)	2
	awareness probability ($AP$)	0.1
HOA	reorganization frequency ($M$)	10
	dominant stallions percent ($DSP$)	10
	horse distribution rate ($HDR$)	10
	horse memory pool ($HMP$)	10
	single stallions percent ($SSP$)	10
	minimum velocity ($V_{min}$)	−0.1
	maximum velocity ($V_{max}$)	0.1
	standard deviation ($sd$)	1
PTA	zero division avoiding constant ($ϵ$)	1.0 × 10300
	fruitiness threshold ($FT$)	0.8
	ripeness threshold ($RT$)	0.2
	minimum fruitiness rate ($F{R}_{min}$)	0.5
	maximum fruitiness rate ($F{R}_{max}$)	1

presents the specific configurations of the nature-inspired algorithms used in the experiments.

The configuration parameters of the PSO algorithm, the GWO, and the CS algorithm were the ones from the EvoloPy library, whereas those of the CSO algorithm, the CSA, and the HOA were taken from the original articles which introduced them.

The configuration parameters of the PTA were selected after a series of in-house experiments. They were inspired by the configuration parameters of other nature-inspired algorithms, especially by those of the CSO algorithm. Therefore, their values were similar to the state-of-the-art values.

Figure 4 presents the boxplot charts for the objective functions $f_1-f_7$ and $g_1-g_7$, and Figure 5 presents the boxplot charts for the objective functions $h_1-h_{10}$.

The PTA showed large variations for $f_3$, $h_8$, and $h_9$. It also showed a larger deviation from the optimal solution in the case of these three objective functions. One of the reasons for this is represented by the values of the configuration parameters. The default values of the PTA configuration parameters were inspired by those used by the algorithms which were its sources of inspiration; this is why it returned overall good results with few exceptions. For example, when FT and RT were set to 0.5 and 0.3, respectively, the results for $f_3$ are 10.76859 as the mean and 15.47429 as the standard deviation (std), the results for $h_8$ are a mean of −10.1531 and a std of 0.000252, and the results for $h_9$ are a mean of −10.4027 and a std of 0.044829.

Figure 6 presents the convergence charts for the objective functions $f_1-f_7$ and $g_1-g_7$, and Figure 7 presents the convergence charts for the objective functions $h_1-h_{10}$.

Table 6. Mean and Standard Deviation Results Summary.

OF	Metric	CSO	PSO	GWO	CS	CSA	HOA	PTA
$f_1$	mean	$4.68\times {10}^{-145}$	$3.03\times {10}^{-11}$	$1.17\times {10}^{-76}$	0.016152	0.01358	0.00551	$3.65\times {10}^{-17}$
	std	$2.209\times {10}^{-144}$	$1.083\times {10}^{-10}$	$4.853\times {10}^{-76}$	0.005253	0.006394	0.001076	$7.068\times {10}^{-17}$
$f_2$	mean	$3.76\times {10}^{-76}$	4.33	$7.75\times {10}^{-45}$	0.79907	1.177365	0.257767	$5.19\times {10}^{-14}$
	std	$1.205\times {10}^{-75}$	5.683171	$1.38\times {10}^{-44}$	0.261191	0.571006	0.025169	$8.3\times {10}^{-14}$
$f_3$	mean	$4.6\times {10}^{-146}$	7.926713	$2.15\times {10}^{-18}$	503.5273	43.52146	0.00389	231.2207
	std	$1.616\times {10}^{-145}$	3.967389	5.223 × 10−18	99.150334	19.19422	0.001535	186.016724
$f_4$	mean	$2.24\times {10}^{-72}$	0.38071	$1.07\times {10}^{-16}$	4.318536	2.821973	0.015582	0.02745
	std	$9.287\times {10}^{-72}$	0.106897	$1.402\times {10}^{-16}$	0.866869	1.038456	0.002264	0.033917
$f_5$	mean	28.90203	45.1364	26.35212	41.3683	85.32425	29.30347	25.69289
	std	0.095961	27.92944	0.737834	11.489354	67.16719	0.095457	0.295923
$f_6$	mean	0	0	0	0.066667	9.7	0	0
	std	0	0	0	0.253708	5.331429	0	0
$f_7$	mean	$2.53\times {10}^{-5}$	4.786226	0.000647	0.034892	0.009394	0.000719	0.005871
	std	$2.9\times {10}^{-5}$	6.627218	0.000304	0.008888	0.002506	0.000304	0.00304
$g_1$	mean	−2567.16	−6489.37	−6296.39	−10249	−7538.05	−3490.74	−9646.43
	std	327.824382	763.8041	984.703141	1397.462624	1428.017	337.816608	933.694813
$g_2$	mean	0	80.30092	2.231542	83.41787	22.72736	1.015421	$3.98\times {10}^{-14}$
	std	0	28.94237	3.596613	8.876992	9.826154	0.185952	$1.259\times {10}^{-13}$
$g_3$	mean	$4.44\times {10}^{-16}$	$4.42\times {10}^{-6}$	$1.41\times {10}^{-14}$	2.150905	3.274961	0.062047	0.665
	std	$5.014\times {10}^{-32}$	$6.533\times {10}^{-6}$	$2.483\times {10}^{-15}$	0.304541	0.853	0.006972	3.64346
$g_4$	mean	0	0.0105	0.002291	0.130508	0.086914	0.000245	$4.1\times {10}^{-12}$
	std	0	0.012776	0.004913	0.056127	0.028191	$5.742\times {10}^{-5}$	$2.169\times {10}^{-11}$
$g_5$	mean	0	$1.52\times {10}^{-9}$	$1.15\times {10}^{-16}$	0.004004	0.042419	$3\times {10}^{-6}$	$7.77\times {10}^{-17}$
	std	0	$5.313\times {10}^{-9}$	$2.026\times {10}^{-17}$	0.002552	0.051366	$6.369\times {10}^{-7}$	$7.796\times {10}^{-17}$
$g_6$	mean	1.239433	$3.05\times {10}^{-13}$	0.019874	1.063431	2.333945	0.75511	0.000138
	std	0.188881	$9.172\times {10}^{-13}$	0.010674	0.510396	1.276191	0.047681	$9.196\times {10}^{-5}$
$g_7$	mean	4.46111	$3.12\times {10}^{-3}$	0.364767	0.329261	0.428364	3.419734	0.013618
	std	0.176022	0.007103	0.182305	0.174058	0.578422	0.087857	0.030512
$h_1$	mean	4.962658	1.85465	3.154627	0.998004	0.998004	1.003432	0.998004
	std	2.798173	1.611231	3.565911	$3.387\times {10}^{-16}$	$4.516\times {10}^{-16}$	0.015257	$4.416\times {10}^{-16}$
$h_2$	mean	0.005439	0.002142	0.004856	0.000322	0.000491	0.000461	0.000431
	std	0.007269	0.004978	0.011963	$3.172\times {10}^{-5}$	0.000372	$6.67\times {10}^{-5}$	0.000279
$h_3$	mean	−1.02244	−1.03163	−1.03163	−1.03163	−1.03163	−1.03162	−1.031628
	std	0.009261	$6.775\times {10}^{-16}$	$6.888\times {10}^{-9}$	$6.775\times {10}^{-16}$	0	$3.407\times {10}^{-6}$	$6.775\times {10}^{-16}$
$h_4$	mean	0.735087	0.397887	0.397888	0.397887	0.397887	0.398699	0.397887
	std	0.432372	$1.129\times {10}^{-16}$	$1.569\times {10}^{-7}$	$1.129\times {10}^{-16}$	$1.129\times {10}^{-16}$	0.001022	$7.455\times {10}^{-15}$
$h_5$	mean	3.299019	3	3.000003	3	3	3.000019	3
	std	0.721271	$1.867\times {10}^{-15}$	$3.739\times {10}^{-6}$	$4.516\times {10}^{-16}$	0	$1.674\times {10}^{-5}$	$1.027\times {10}^{-14}$
$h_6$	mean	−3.6737	-3.86278	−3.86251	−3.86278	−3.86278	−3.8419	−3.861731
	std	0.12806	$1.355\times {10}^{-15}$	0.001192	$1.355\times {10}^{-15}$	$2.258\times {10}^{-15}$	0.013793	0.002725
$h_7$	mean	−2.38375	−3.25157	−3.2315	−3.322	−3.27444	−2.87276	−3.221027
	std	0.328454	0.087861	0.062884	$3.834\times {10}^{-12}$	0.059241	0.111650	0.096315
$h_8$	mean	−1.24178	−7.95926	−9.30929	−10.1532	−10.1532	−3.28108	−8.720348
	std	0.662518	2.551791	1.91861	$2.569\times {10}^{-14}$	0	1.777358	2.453807
$h_9$	mean	−1.34943	−9.26524	−10.4026	−10.4029	−10.1484	−2.99821	−8.743973
	std	0.460402	2.348016	0.000193	$1.656\times {10}^{-13}$	1.394326	1.018029	2.846614
$h_{10}$	mean	−1.47508	−10.0003	−10.5361	−10.5364	−10.5364	−3.6214	−10.17745
	std	0.525267	1.635722	0.000166	$1.827\times {10}^{-12}$	0	1.11755	1.366066

presents the mean and the std results for each of the objective functions $f_1-f_7$, $g_1-g_7$, and $h_1-h_{10}$.

Table 7. Rank Comparison Results Summary.

Objective Functions	Metric	CSO	PSO	GWO	CS	CSA	HOA	PTA
$f_1-f_7$	mean rank	1.28571	4.85714	1.85714	6.14285	6	3.28571	3.14285
	rank	1	5	2	7	6	4	3
$g_1-g_7$	mean rank	3.42857	3.42857	3.57142	5	5.71428	4.42857	2.42857
	rank	2	2	4	6	7	5	1
$h_1-h_{10}$	mean rank	7	3.1	3.9	1	1.6	5.5	3.3
	rank	7	3	5	1	2	6	4
all	mean rank	4.29166	3.70833	3.20833	3.66666	4.08333	4.54166	3
	rank	6	4	2	3	5	7	1

presents the summarized comparison of the results from Table 6 in terms of performance ranking in which better ranks are assigned to the best solutions by using an approach like the one presented in [96]. The best rank was considered to be 1, and in the cases in which two solutions had the same value, they were assigned the same rank. The last row of the table presents the results when the same index computation, namely, the mean rank and the rank, was applied for all 24 benchmark objective functions.

As shown in Table 7, the PTA returned results comparable or even better than the ones returned from the other nature-inspired algorithms. It ranked third for $f_1-f_7$, first for $g_1-g_7$, and fourth for $h_1-h_{10}$. When the mean rank and the rank were computed for all objective functions, the PTA had the best rank.

Different experiments with other values for the configuration parameters showed better results for the PTA for some objective functions. When FT and RT were set to 0.5 and 0.3, respectively, the PTA ranked second for the functions $h_1-h_{10}$. However, the differences were insignificant for the other two groups, namely, $f_1-f_7$ and $g_1-g_7$, for which it ranked third and first, respectively, as when the default values were used.

The PTA was designed to return good results across all 24 benchmark objective functions rather than to return the best results for each group of functions; this is the principal reason why it did not return the best results for $f_1-f_7$ and $h_1-h_{10}$ under the default configurations.

5. PTA and Machine Learning Ensembles for Energy Efficiency Estimation

This section is organized into four subsections. The first subsection presents the data used as experimental support, namely, the Energy Efficiency Dataset from the UCI Machine Learning repository. The second subsection presents the PTA application for the machine learning ensembles for the estimation of the energy efficiency. The third subsection presents the data standardization adaptation of the PTA ensemble-based energy efficiency estimation methodology. The fourth subsection presents the energy efficiency estimation results.

5.1. Energy Efficiency Dataset Description

The Energy Efficiency Dataset is characterized by 768 instances, 8 attributes, and 2 responses. The data was obtained using 12 building shapes, which were simulated in Ecotect.

Table 8. Energy Efficiency Dataset Attribute Information Summary.

Column Name	Variable Name	Range
X1	Relative Compactness	[0.62, 0.98]
X2	Surface Area	[514.5, 808.5]
X3	Wall Area	[245, 416.5]
X4	Roof Area	[110.25, 220.5]
X5	Overall Height	[3.5, 7]
X6	Orientation	[2, 5]
X7	Glazing Area	[0, 0.4]
X8	Glazing Area Distribution	[0, 5]
Y1	Heating Load	[6.01, 43.1]
Y2	Cooling Load	[10.9, 48.3]

presents the summary of the attributes’ information.

The features X1–X8 describe the attributes, whereas the features Y1–Y2 describe the responses.

5.2. PTA for Ensemble Weight Optimization for Energy Efficiency Estimation

Figure 8 presents the PTA methodology for ensemble weight optimization used in the estimation of energy efficiency.

The steps of the methodology are as follows:

 Step 1.

The input is the Energy Efficiency Dataset. The data samples of the original dataset are shuffled randomly. Two datasets are derived from this dataset, namely, the Cooling Dataset and the Heating Dataset, depending on the column used for the prediction. The Cooling Dataset does not contain the Y1 column, whereas the Heating Dataset does not contain the Y2 column.

 Step 2.

In the cross-validation phase, fivefold cross validation is used.

Table 9. Cooling Dataset and Heating Dataset Folds Summary.

Dataset	Fold	Number of Samples
Cooling Dataset	fold 1	154
	fold 2	154
	fold 3	154
	fold 4	153
	fold 5	153
Heating Dataset	fold 1	154
	fold 2	154
	fold 3	154
	fold 4	153
	fold 5	153

presents a summary of the number of samples of each fold.

 Step 3.

The Training Data and the Testing Data are represented by 80% and 20% of the total samples, respectively. In each of the five cases, the Testing Data is represented by one distinct fold from the five folds, whereas the Training Data is represented by the remaining four folds.

 Step 4.

The four algorithms of the ensemble are the Random Forest Regressor (RFR), the Gradient Boosting Regressor (GBR), the AdaBoost Regressor (AdaBoost), and the Extra Trees Regressor (ETR). The implementations of these regressors are the ones from the sklearn.ensemble library, and their default configurations are used. The only parameter which is overwritten was the random_state, which is set to an arbitrary constant, namely 42, to obtain reproducible research results.

 Step 5.

The original PTA is adapted to compute the weights of the ensemble. The number of dimensions of the search space $D$ is set to 4, a value equal to the number of the algorithms of the ensemble.

Then, for each plum P_i, where $i$ is the index of the plum, the partial weights are computed using the following formula:

$$\left[\begin{array}{c}{w}_{RFR}^{\prime }\\ w_{GBR}^{\prime }\\ w_{AdaBoost}^{\prime }\\ w_{ETR}^{\prime }\end{array}\right]=\left[\begin{array}{c}{P}_{i,1}+X_{max}\\ P_{i,2}+X_{max}\\ P_{i,3}+X_{max}\\ P_{i,4}+X_{max}\end{array}\right],$$

(8)

whereas the ensemble weights are computed using the following formula:

$$\left[\begin{array}{c}{w}_{RFR}\\ w_{GBR}\\ w_{AdaBoost}\\ w_{ETR}\end{array}\right]=\{\begin{array}{cc}\left[\begin{array}{c}\frac{w_{RFR}^{\prime }}{w_{RFR}^{\prime }+w_{GBR}^{\prime }+w_{AdaBoost}^{\prime }+w_{ETR}^{\prime }}\\ \frac{w_{GBR}^{\prime }}{w_{RFR}^{\prime }+w_{GBR}^{\prime }+w_{AdaBoost}^{\prime }+w_{ETR}^{\prime }}\\ \frac{w_{AdaBoost}^{\prime }}{w_{RFR}^{\prime }+w_{GBR}^{\prime }+w_{AdaBoost}^{\prime }+w_{ETR}^{,}}\\ \frac{w_{ETR}^{,}}{w_{RFR}^{\prime }+w_{GBR}^{\prime }+w_{AdaBoost}^{\prime }+w_{ETR}^{,}}\end{array}\right],& \mathrm{if} w_{RFR}^{\prime }+w_{GBR}^{\prime }+w_{AdaBoost}^{\prime }+w_{ETR}^{\prime }\ne 0,\hfill \\ \hfill \left[\begin{array}{c}0.25\\ 0.25\\ 0.25\\ 0.25\end{array}\right],& \text{otherwise}.\hfill \end{array}$$

(9)

Considering the values of $y_{RFR}$, $y_{GBR}$, $y_{AdaBoost}$, and $y_{ETR}$ predicted by using the RFR, GBR, AdaBoost, and ETR, respectively, as well as the values $z$ of the Standardized Testing Data, and the number of samples $NS$ of the Standardized Testing Data, the objective function $OF$ has the following formula:

$$OF=\frac{1}{2\times NS}{{\displaystyle \sum }}_{j=1}^{NS}{\left(z_j^{\prime }-z_j\right)}^2,$$

(10)

such that:

$$z_j^{\prime }=w_{RFR}^{\prime }\times y_{RFR,j}+w_{GBR}^{\prime }\times y_{GBR,j}+w_{AdaBoost}^{\prime }\times y_{AdaBoost,j}+w_{ETR}^{\prime }\times y_{ETR,j}, \mathrm{where} j=\overline{1,NS}.$$

(11)

 Step 6.

The optimized ensemble weights are the ones which are computed from the $p_{gbest}$ value.

 Step 7.

The ensemble is evaluated using the Root-Mean-Square Error (RMSE), R-squared ($R^2$), the Mean Absolute Error (MAE), and the Mean Absolute Percentage Error (MAPE), represented by Formulas (12)–(15), respectively:

$$RMSE=\sqrt{\frac{{{\displaystyle \sum }}_{j=1}^{NS}{\left(z_j-y_j\right)}^2}{NS}},$$

(12)

$$R^2=1-\frac{{{\displaystyle \sum }}_{j=1}^{NS}{\left(z_j-y_j\right)}^2}{{{\displaystyle \sum }}_{j=1}^{NS}{\left(z_j-\overline{y}\right)}^2},$$

(13)

$$MAE=\frac{{{\displaystyle \sum }}_{j=1}^{NS}\left|z_j-y_j\right|}{NS},$$

(14)

$$MAPE=\frac{1}{NS}\times {{\displaystyle \sum }}_{j=1}^{NS}\frac{\left|z_j-y_j\right|}{z_j},$$

(15)

such that $y$ represents the values predicted by the ensemble, and $\overline{y}$ is defined as:

$$\overline{y}=\frac{{{\displaystyle \sum }}_{j=1}^{NS}{y}_j}{NS}.$$

(16)

5.3. Data Standardization Adaptation

The PTA methodology presented in Figure 8 was adapted such that the training data and the testing data were standardized to obtain better predictability. The z-score was used for the standardization of the Training Data, the Testing Data was standardized using the values of the mean and the standard deviation computed in the standardization of the Training Data, and the resulting Standardized Training Data and Standardized Testing Data were used as input for the components of the ensemble and in the evaluation of the performance of the predictions. The resulted standardized data were called the Standardized Cooling Dataset and the Standardized Heating Dataset, respectively.

5.4. Energy Efficiency Estimation Results

The energy efficiency estimation experiments were performed using the sklearn package. The values of the specific configuration parameters of the PTA were the default ones presented in Table 5.

Table 10. PTA-Adapted Parameters for Ensemble Weight Optimization Summary.

Parameter	Value
number of iterations ($I$)	500
number of dimensions ($D$)	4
number of plums (N)	50
maximum possible position value ($X_{max}$)	2000
minimum possible position value ($X_{min}$)	−2000
objective function ($OF$)	formula (10)

summarizes the adapted parameters of the PTA for the optimization of the ensemble weights.

For each dataset used in the experiments, namely, the Cooling Dataset and the Heating Dataset, and for each configuration resulting from the cross-validation operation, 30 runs were performed. The total number of runs was equal to $2\times 5\times 30=300$.

The std values were either zero or a very small number for both the Cooling Dataset and the Heating dataset; therefore, the mean values of the RMSE, R², MAE, and MAPE were also the best ones.

Figure 9 presents the summary of the mean RMSE results.

The best mean results for the Cooling Dataset and the Heating Dataset were returned for fold 4 and fold 1, respectively. Another remark is that the mean RMSE results were much better for the Heating Dataset for all five folds.

Figure 10 presents the detailed results summary that corresponds to the runs which resulted in the best RMSE values. The values for the $R^2$, MAE, and MAPE were equal in each of the 30 runs for each fold; therefore, the results were the same when the best values of these indicators were used as selection criteria for the most representative runs.

The detailed results for each fold show that the $R^2$ prediction results were comparable for the two datasets for each fold, which means that the PTA ensembles’ optimization models fit the data well for both datasets and for all folds. The best MAE and MAPE results were obtained for the Heating Dataset for all folds.

Figure 11 presents a comparison of the running times of the runs which resulted in the best RMSE values.

The running time results varied more for the Heating Dataset than for the Cooling Dataset. The differences are justified by the different durations required to train each regressor of the ensemble.

Figure 12 presents the weight results for each algorithm of the ensemble for each fold.

As can be seen in the figure, the PTA-optimized ensemble weights varied more in the case of the Heating Dataset than in the case of the Cooling Dataset. Although the GBR received the highest weight for each fold, the AdaBoost received a weight equal to zero for each fold of the Cooling Dataset. On the other hand, the ETR received the highest weight for three out of the five folds, whereas AdaBoost received non-zero weights for fold 4 and fold 5 of the Heating Dataset.

6. Discussions

This section compares the results obtained using the PTA-based method considering the following performance factors: (1) the application of the data standardization procedure, (2) the results obtained by the average ensemble and the individual components of the ensemble, and (3) the results obtained by other nature-inspired algorithm-based ensembles. Finally, the last subsection presents a comparison to representative results obtained in other research studies.

6.1. Data Standardization Results

The application of data standardization led to much better results in terms of RMSE. Figure 13 presents the detailed results summary for the runs which led to the best RMSE values. The results were almost identical when other indicators were considered as selection criteria for the most representative runs.

The prediction results were much better for the Standardized Heating Dataset than for the Standardized Cooling Dataset for each fold, as shown in the Heating Dataset and Cooling Dataset energy prediction results. The values of $R^2$ were close to 1, which means that predictability performance was similar for both standardized datasets. Regarding running time, the average running times for the best runs for the Standardized Cooling Dataset and Standardized Heating Dataset were 29.35312 s and 24.94228 s, respectively. These values are slightly higher than when no standardization was applied, which can be justified by the time required to standardize the data and the way in which the algorithms behave when the data is standardized.

Figure 14 presents the weights of each algorithm of the ensemble for each fold when data standardization is applied. These weights are the ones which correspond to the best runs in terms of RMSE.

The weights of the ensemble for the Standardized Cooling Dataset are almost identical to the weights obtained for the Cooling Dataset. Even though the weights were similar for fold 1, fold 3, and fold 5 for the Standardized Heating Dataset and the Heating Dataset, there were significant differences between fold 2 and fold 4. However, the GBR and the ETR obtained the highest weights of the ensemble for both fold 2 and fold 4.

6.2. Comparison to Average Ensemble and Individual Components of the Ensemble

Table 11. Comparison to the Average Ensemble and the Individual Components of the Ensemble.

Dataset	Approach	RMSE	R2	MAE	MAPE
Cooling	PTA ensemble	1.519352	0.974118	1.016108	0.03663
	average ensemble	1.712945	0.96694	1.13183	0.040357
	RFR	1.758939	0.964869	1.050144	0.036053
	GBR	1.523731	0.973964	1.028088	0.037196
	AdaBoost	2.44213	0.933135	1.897533	0.075278
	ETR	1.892635	0.959707	1.067652	0.036198
Heating	PTA ensemble	0.433806	0.998119	0.296516	0.012694
	average ensemble	0.660918	0.995658	0.490883	0.023875
	RFR	0.481654	0.997662	0.328409	0.014023
	GBR	0.477339	0.997731	0.340544	0.01553
	AdaBoost	1.892841	0.964484	1.529491	0.077095
	ETR	0.45755	0.997912	0.308258	0.01284
Standardized Cooling	PTA ensemble	0.159903	0.974098	0.107013	0.169855
	average ensemble	0.179606	0.967172	0.119465	0.212976
	RFR	0.184274	0.965269	0.109801	0.204687
	GBR	0.160374	0.973942	0.108143	0.167695
	AdaBoost	0.252388	0.935285	0.198629	0.282862
	ETR	0.200184	0.959175	0.113029	0.21786
Standardized Heating	PTA ensemble	0.043124	0.998111	0.02945	0.031894
	average ensemble	0.066474	0.995548	0.049229	0.058983
	RFR	0.048122	0.997628	0.03277	0.038727
	GBR	0.047517	0.997717	0.033849	0.03378
	AdaBoost	0.189049	0.963962	0.151101	0.16832
	ETR	0.045553	0.997896	0.030651	0.033626

compares the best results obtained by the PTA ensemble to the ones obtained by the average ensemble and by the algorithms of the ensemble. The evaluation metrics presented in the table were computed as the averages of those obtained for each fold.

As shown in the table, the PTA ensemble returned the best RMSE results for each dataset compared to the other approaches, which were based on the average ensemble and on the individual components of the ensemble, namely, the RFR, the GBR, the AdaBoost, and the ETR. For each dataset, the PTA ensemble returned the R² values which were the closest to one, which means that it fit the data the best. Regarding the MAE values, it also returned the best results. However, in the case of the MAPE values, there were exceptions for the Cooling Dataset, in which the values returned by the RFR and the ETR were better, and for the Standardized Cooling Dataset, in which the value returned by the GBR was better. The differences were very small or insignificant. These differences can be justified by the objective function used by the ensemble, which gave more attention to the minimization of the RMSE.

6.3. Comparison to Other Nature-Inspired Algorithm-Based Ensembles

This section compares the performance of the PTA ensemble to the performance of the ensembles based on other nature-inspired algorithms both for when no standardization was considered and for when the data was standardized.

For each nature-inspired algorithm considered in the experiments, the same specific configuration parameters values were applied as the ones presented in Table 4. For each algorithm and for each fold, 30 runs were performed. The results did not present a lot of differences during the runs; therefore, the best runs were the ones for which the RMSE value was minimal. If more runs returned the same minimal RMSE value, then the run that corresponded to the minimum running time was selected. For each dataset, the mean values of the evaluation metrics were computed as averages of the values which corresponded to each fold.

The values of RMSE, $R^2$, MAE, MAPE, and of the weights of the ensembles were almost identical for all 30 runs for all algorithms, except for the CSO algorithm and the PSO algorithm. For example, for the CSO–based ensemble, the RMSE had values from [1.394474, 1.47675] for fold 1 of the Cooling Dataset. Similarly, for the PSO-based ensemble, the RMSE had values from [0.3438, 0.3522] for fold 1 of the Heating Dataset. However, the best runs out of the 30 runs for both the PSO and CSO algorithms were similar or identical to the ones of the other nature-inspired algorithms, except for the running time.

Figure 15 presents the summary of the average running time for all nature-inspired algorithm-based ensembles considered in the experiments, with separate graphs for each experimental dataset showing the runs that returned the best RMSE values.

As shown in Figure 15, the standardization of the data did not greatly influence the average running times of the nature-inspired algorithms. The worst performance was that of the HOA, whereas the best performance was that of the GWO for the Cooling Standardized Dataset and that of the CSO algorithm for the other three datasets.

As an overall remark, from the perspective of running time and the variation in the results, the GWO-based ensemble returned the best results. The PTA performed quite well; it was similar to the CS and CSA in terms of running time and similar to the GWO, the CS algorithm, the CSA, and the HOA in terms of variation in the results.

Finally, the PTA results in terms of RMSE for the best runs for all four datasets were equal to the ones returned by the GWO, the CS algorithm, and the CSA. The effectiveness of the PTA is justified by the fact that it was not outperformed by the other nature-inspired algorithms in terms of RMSE, and the returned results varied insignificantly for all 30 runs for each fold of each dataset. These results show that the PTA is a reliable algorithm which can be used in other engineering applications as well.

6.4. Comparison to the Results from Other Research Studies

This section compares the results obtained in this study to those obtained by other representative research studies which used the same dataset and the same cross-validation type, namely, fivefold cross-validation. The CS algorithm, the CSA, the GWO, and the PTA ensemble-based methods which also considered the standardization of the data are abbreviated as CS-S, CSA-S, GWO-S, and PTA-S, respectively.

Table 12. Comparison to the Literature Results.

Year	Approach	Description	Method	Cooling (RMSE)	Heating (RMSE)
2023	this	A method based on a PTA ensemble of RFR, GBR, AdaBoost, and ETR is compared to the performance of other nature-inspired algorithm-based ensembles.	CS, CSA, GWO, PTA	1.519352	0.433806
			CS-S, CSA-S, GWO-S, PTA-S	0.159903	0.043124
2022	Ghasemkhani et al. [97]	A method based on the Tri-Layer Neural Network (TNN) and Maximum Relevance Minimum Redundancy (MRMR) is compared to other prediction methods.	TNN and MRMR	0.81391	0.45689
			GPR	1.31090	0.46479
			Boosted Trees	1.63640	0.66731
			Fine Tree	1.80640	1.02460
			Bagged Trees	1.99780	0.79549
			Medium Tree	1.87700	1.11810
			SLR	1.93530	1.09030
			LR	2.30970	2.15290
			SVM	2.19510	2.32150
			Coarse Tree	2.61780	2.94610
2019	Prasetiyo et al. [98]	A method based on the Naïve Bayes Classifier (NBC).	NBC	0.1412	0.1491
2019	Bui et al. [99]	A method in which the weights and the biases of an ANN are optimized using the GA and the Imperialist Competition Algorithm (ICA).	ANN	3.9489	3.6481
			GA-ANN	2.8598	2.8878
			ICA-ANN	2.7795	2.7819

summarizes the comparison of the results.

The comparison to the results from [97], in which methods based on the Tri-Layer Neural Network (TNN) and the Maximum Relevance Minimum Redundancy (MRMR), GPR, Boosted Trees, Medium Tree, Fine Tree, Bagged Trees, LR, SVM, Stepwise Linear Regression (SLR), and Coarse Tree were used, is not accurate even though the same ratio for cross-validation was used because that approach also had a data preprocessing phase in which the irrelevant, noisy, and redundant data was eliminated, and it also had a feature selection phase in which the most representative features were selected. The results are comparable to those obtained by the CS algorithm, CSA, GWO, and PTA methods. The nature-inspired algorithm-based ensemble methods returned better RMSE values than all of the other approaches for the Heating Dataset; however, they were outperformed in the Cooling Dataset by the TNN and MRMR and by the GPR.

Compared to the method presented in [98], where a Naïve Bayes Classifier was applied, the CS-S, CSA-S, GWO-S, and PTA-S methods were better in terms of heating prediction results, but they did not lead to better results in terms of cooling prediction results.

The authors of [99] considered the application of the GA and the Imperialist Competition Algorithm (ICA) [100] in the optimization of the biases and the weights of an ANN. The results showed that the optimization of the ANN using metaheuristics led to better results. The results obtained by the nature-inspired algorithm-based methods described in this article were better, but the results returned by the ANN methods showed fewer differences between the Cooling Dataset and Heating Dataset results.

7. Conclusions

This article presents a novel approach based on the PTA for the estimation of energy efficiency in which the weights of an ensemble composed of the RFR, the GBR, the AdaBoost, and the ETR is used in the prediction of cooling and heating loads. Fivefold cross-validation was used to validate the method, and the results obtained for the cooling and heating predictions were 1.519352 and 0.433806, respectively, when no standardization was applied, and they were 0.159903 and 0.043124, respectively, when standardization was applied. The obtained results were also compared to those in the literature.

The principal outcomes are the following:

• The PTA is a performant nature-inspired algorithm, as shown by the comparison to six other nature-inspired algorithms using 24 benchmark objective functions;
• The PTA has fewer configuration parameters compared to the HOA and CSO algorithms, which makes it easier to configure for certain optimization problems;
• On the other hand, the PTA has more configuration parameters than the GWO, the CS algorithm, and the CSA, which might be an advantage for other types of optimization problems;
• The PTA and PTA-S ensembles showed better prediction results in terms of RMSE, $R^2$, MAE, and MAPE compared to the average ensemble and the individual components of the PTA-based ensembles;
• The quantitative results obtained by using the PTA and PTA-S methods to predict energy efficiency were equal to those obtained by using the CS algorithm, the CSA, the GWO, the CS-S, the CSA-S, and the GWO-S;
• The results obtained by using the nature-inspired algorithm-based methods for the prediction of energy efficiency are comparable to those obtained in the literature with state-of-the-art methods.

As future research work, the following directions are proposed:

• The evaluation of the PTA using more objective benchmark objective functions and the comparison to other nature-inspired algorithms;
• The improvement of the performance of the PTA through hybridization with other nature-inspired algorithms or the modification of the equations applied to update the positions of the flowers;
• The application of the PTA to more types of engineering problems;
• The development of a multi-objective version of the PTA.