Sensitivity Assessment of Building Energy Performance Simulations Using MARS Meta-Modeling in Combination with Sobol’ Method

Abstract

Large discrepancies can occur between building energy performance simulation (BEPS) outputs and reference data. Uncertainty and sensitivity analyses are performed to discover the significant contributions of each input parameter to these discrepancies. Variance-based sensitivity analyses typically require many stochastic simulations, which is computationally demanding (especially in the case of the large number of input parameters involved in the analysis). To overcome these impediments, this study proposes a reliable meta-model-based sensitivity analysis, including validation, Morris’ method, multivariate adaptive regression splines (MARS) meta-modeling, and Sobol’ method, to identify the most influential input parameters on BEPS prediction (annual energy consumption) at the early building design process. A hypothetical building is used to analyze the proposed methodology. Six statistical metrics are applied to verify and quantify the accuracy of the model. It is concluded that the cooling set-point temperature and g-value of the window are the most influential input parameters for the analyzed case study.

1. Introduction

Over the past decades, building energy performance simulation (BEPS) has been particularly popular as a major approach to estimating the energy consumption of buildings and, subsequently, reducing the accompanying greenhouse gas emissions. However, the energy performance of buildings depends on a vast number of factors, including the building’s characteristics, HVAC systems, control and maintenance, weather conditions, sociological factors, and occupants’ behavior [1,2]. Due to the uncertainty in these parameters of a model, huge and sometimes serious discrepancies can arise between BEPS predictions provided by different dynamic BEPS tools and reference data. This mismatch is commonly referred to as the “Energy Performance Gap,” which can be classified into three different groups: the gap between first-principle energy models and measurements, the gap between machine learning approaches and measurements, and the gap between predictions and display certificates in legislation [3,4].

Due to the proliferation of numerous BEPS tools, an acceptable degree of confidence in BEPS prediction has become a salient preoccupation for the building simulation community. Validation procedures help to ensure the accuracy and plausibility of the simulation results. The purpose of the validation is to evaluate the quality and reliability of the predicted and calculated performance simulations and to identify errors in the simulation outputs. Subsequently, by using calibration approaches, the uncertainties can be reduced or eliminated, and eventually, simulation results can be aligned with reference data to the greatest extent. Three different kinds of techniques are applied to the validation of dynamic thermal models of buildings: empirical validation, analytical verification, and software-to-software comparative testing [5,6,7].

To quantify the accuracy of building models, i.e., to characterize the discrepancies between simulation results and real data, statistical performance metrics are used. There are several types of statistical metrics that have been commonly used in the literature [8,9,10,11,12,13,14,15,16,17,18,19,20,21,22], including the Mean Absolute Error (MAE), Mean Bias Error (MBE), Normalized Mean Bias Error (NMBE), Root Mean Squared Error (RMSE), Coefficient of Variance of Root Mean Squared Error (CV-RMSE), and Coefficient of Determination (R²). A large number of previous studies have used the mentioned validation techniques and statistical criteria to evaluate the level of prediction accuracy of energy performance models [23,24,25,26,27,28,29,30,31,32,33].

To discover the most influential building model parameters on the simulation outputs (i.e., parameters that contribute enormously to the discrepancies between the energy performance simulation output and the reference data), uncertainty and sensitivity analyses are conducted [34,35,36,37,38].

Table 1. Validation methods, sensitivity analysis, and statistical performance metrics found in the literature for the assessment of BEPS, where ‘X’ indicates methods/metrics involved in the reviewed literature.

Reference	Year	Validation Methods			Sensitivity Analysis	Metrics
		Empirical	Comparative	Analytical		MBE	NMBE	RMSE	CVRMSE	R2	MAE	Other
Judkoff et al. [7]	2008	X	X	X	X
Loutzenhiser et al. [23]	2009	X	X		X						X	X
Tian et al. [32]	2009		X				X		X
Raslan et al. [39]	2010		X
Shrestha et al. [40]	2011	X
Tabares-Velasco et al. [41]	2011		X	X
Antretter et al. [42]	2011	X	X
Tabares-Velasco et al. [24]	2012	X	X	X				X
Spitz et al. [35]	2012	X			X
Zhu et al. [43]	2013		X
Kim et al. [44]	2013		X		X							X
Lauster et al. [8]	2014	X		X				X		X
Mateus et al. [25]	2014	X			X	X				X		X
Strachan et al. [36]	2015	X			X
Roberti et al. [9]	2015	X			X			X			X
Alaidroos and Krarti [26]	2016	X				X		X	X
Sandels et al. [27]	2016	X						X		X
Ferroukhi et al. [10]	2016	X						X				X
Buonomano [45]	2016		X
Moronis et al. [46]	2017			X								X
Cattarin et al. [28]	2018	X	X					X				X
Cattarin et al. [37]	2018	X			X
Zhang et al. [11]	2018	X					X		X
Nageler et al. [29]	2018	X	X			X		X				X
Kim et al. [12]	2019	X					X		X
Lee and Hong [13]	2019	X					X		X
Barone et al. [30]	2019	X	X								X	X
Im et al. [14]	2019	X					X		X
Halimov et al. [15]	2019	X	X	X				X		X
Kalogeras et al. [16]	2020		X		X			X		X		X
Gutiérrez et al. [17]	2020	X	X				X		X	X	X	X
Eriksson et al. [18]	2020	X			X		X		X	X
Nouri et al. [47]	2020		X
Shi and Chen [19]	2021	X	X				X		X		X	X
Yi et al. [20]	2021	X					X		X
Alongi et al. [38]	2021	X		X	X
Catto Lucchino et al. [33]	2021	X	X			X	X	X	X
Nouri et al. [31]	2021		X			X	X	X	X	X
Zakula et al. [48]	2021		X		X			X	X			X
Eguía-Oller et al. [21]	2021	X					X	X	X		X
Magni et al. [22]	2022		X		X	X	X		X		X	X

summarizes the validation methods, sensitivity analysis and statistical performance indices used in the reviewed literature in the context of BEPS.

Sensitivity analysis (SA) methods may be categorized into two major groups: local sensitivity analysis (LSA) and global sensitivity analysis (GSA) [49,50]. LSA is based on one-factor-at-a-time (OAT) approaches, where the inference about the model output variation is made by varying only one model input parameter at a time, and all other parameters are kept constant at their nominal values (or baseline points). Although LSA is computationally efficient [51] and easy to comprehend, it is predicated on the assumption that the model is linear (or at least additive) [52]. Furthermore, it cannot grasp mutual interactions between input parameters and is not feasible to explore multidimensional parameter spaces [49]. Therefore, the robustness and reliability of LSA cannot be guaranteed.

The GSA, on the other hand, investigates the model output where all the input parameters chosen for analysis are varied over the whole parameter space simultaneously. Therefore, GSA imparts further information regarding the correlation between input parameters and may be more reliable. Various methods have been proposed for GSA. The most commonly used approaches are a regression method [53], a screening-based method (such as Morris’ method) [34,54,55], and a variance-based method (such as the Sobol’ method and Fourier Amplitude Sensitivity Test (FAST)) [35,48,55,56]. Due to the non-linear, non-monotonic behavior of sophisticated building models, GSA has been widely implemented to rank the most important parameters [55,57]. However, the greatest shortcoming of GSA lies in its computationally demanding evaluations, especially in the early stages of the building design process, in which many design parameters are explored and a quick evaluation of the building performance is highly desirable. To implement GSA methods, a large number of stochastic simulations are typically required. For computationally expensive models, comprehensive GSA can be time-intensive and hence increase their cost [51,58].

To tackle the computational burdens of GSA for intricate models such as those used in dynamic BEPS (i.e., physics-based models), this study proposes a metamodeling-based SA method. Meta-models (also referred to as surrogate models or emulators) are approximate, fast models that can be substituted for highly sophisticated, time-consuming models (such as detailed engineering-based energy models) with acceptable reliability [59,60,61]. Thus, for a large number of simulation runs, meta-models are less computationally demanding in comparison with original, complex building models. A variety of statistical meta-modeling approaches have been employed in the context of BEPS, including regression analysis [62,63,64], support vector regression (SVR) [65,66,67], multivariate adaptive regression splines (MARS) [59,68,69], gaussian process regression (GPR) [60,61], Random Forest (RF) [69], neural networks [59,62], and ensemble models [63].

Rodríquez et al. [70] applied a regression method to calculate sensitivity indices using a case study of a dwelling with three ranges of weather data and three levels of occupancy. A sample size of 200 was investigated to create the set of parameters. Bucking et al. [71] proposed a methodology to identify the influential variables (design and operation parameters) on the building’s performance (annual electricity consumed). A back-tracking search explored and ranked 8 of 26 variables that have a significant impact on the energy consumption in a net-zero energy house using 6000 EnergyPlus simulation runs. Kim [72] developed two meta-models (gaussian process emulator and polynomial chaos expansion) in MATLAB to alleviate the computational burden of BEPS tools, using a five-story office building located in South Korea. The uncertainty results were propagated by 200 simulation runs using the EnergyPlus model, which has 47 input parameters. Hester et al. [64] presented a probabilistic framework using a regression-based energy meta-model to estimate the energy consumption of single-family buildings at the early stages of the design process. A total of 17 building attributes were selected, and Monte Carlo simulations were carried out to evaluate the variability in the meta-model output (energy consumption).

An extensive review of validation, uncertainty, and sensitivity analysis studies in the context of BEPS reveals several substantial research gaps. The choice of uncertain parameters for performing the SA is often based on either expert knowledge or existing previous studies, which can seriously impede an accurate and reliable SA. Furthermore, running stochastic simulations with a large number of samples in BEPS tools is very time-consuming or even impractical in the case of complex models. Finally, the application of the MARS meta-model in performing variance-based SA (in particular the Sobol’ method) has rarely been carried out in the literature, and further studies are required. This study proposes a novel, four-stage SA framework, including validation, Morris’ method, meta-modeling, and Sobol’ method. The proposed framework improves the robustness and accuracy of the SA in the early building design process, considering a large set of uncertainties (105 uncertain parameters), to support robust decision-making under input uncertainties. The MARS machine learning algorithm is applied to develop a meta-model for the complex building model, and its prediction accuracy is scrutinized by comparison with three additional meta-models, namely Polynomial Regression (PR), Random Forest (RF), and Support Vector Regression (SVR) based on the radial basis function (RBF) kernel, using several statistical metrics.

The proposed methodology is applied to answer the main research question of this study: Which uncertain input parameters (in the analyzed case) drive the majority of the model output variation? The remainder of this paper is structured as follows: Section 2 introduces the proposed methodology. In Section 3, the investigated case study, the simulation tool used, and the weather data are presented. The results and discussion are described in Section 4. Finally, Section 5 gives concluding remarks.

2. Methodology

The methodology in this study is comprised of two constituent parts. The first part involves the validation approach, which is applied to evaluate and quantify the accuracy of dynamic BEPS results.

Two-time series datasets are created: reference and synthetic datasets. The reference dataset is built from the simulation results of the investigated case study (see Section 3) for one year, containing values of heating and cooling loads at an hourly resolution. To conduct the validation and evaluate the discrepancies, a virtual building is modeled. The virtual building uses the same simulation setup; however, the values of the input parameters, i.e., the thermophysical and optical properties of the building envelope, infiltration rate, internal heat gains, and HVAC system, are randomly varied using Latin Hypercube sampling (LHS). For each set of input parameters, the probability density function is given as a uniform distribution. The simulation results of the virtual building (heating and cooling loads) are collected in a synthetic dataset. The major advantage of the synthetic dataset over the measurement dataset is that there are no measurement errors, and all required information regarding the parameters of the building is attainable.

The main aim of creating the synthetic dataset is to assess the deviation of the simulation results and, subsequently, to inspect the most influential input parameters on the simulation outputs by SA, which will be discussed in the second part of the methodology.

Both qualitative and quantitative validations are performed. In quantitative validation, statistical metrics are applied to ascertain and quantify the mismatch between the simulation results of reference and synthetic datasets (see Section 2.1). The validation approach used in the first part of the methodology is schematically illustrated in Figure 1.

The second part of the methodology investigated in this study presents the uncertainty and sensitivity analysis, consisting mainly of three sequential stages: screening-based SA (Morris’ method), MARS meta-modeling, and variance-based SA (Sobol’ method).

First, the screening-based SA by Morris [73] is performed to determine important input parameters and reduce the number of parameters that are later used in the metamodel-based SA. Since this study investigates a large number of parameters, implementing Morris’ method prior to performing variance-based SA (Sobol’ method) can substantially reduce the computational cost by selecting key parameters influencing BEPS results. Note that in this study, the term “computational cost” is interpreted as the number of times the investigated model has to be evaluated. 105 uncertain input parameters, including thermophysical properties, infiltration rate, internal heat gains, and HVAC system, are selected.

Table A1. List of input parameters and assigned uncertainty ranges used for the screening-based sensitivity analysis by Morris’ method.

Index	Parameter	Baseline Point	Parameter Range (Lower and Upper Bounds)
	Wall south
1	Conductivity layer 1 (plaster)	0.140 W/(m·K)	[0.126, 0.154]
2	Density layer 1 (plaster)	800 kg/m3	[720, 880]
3	Specific heat capacity layer 1 (plaster)	1500 J/(kg·K)	[1350, 1650]
4	Thickness layer 1 (plaster)	0.005 m	[0.0045, 0.0055]
5	Conductivity layer 2 (insulation)	0.040 W/(m·K)	[0.036, 0.044]
6	Density layer 2 (insulation)	50 kg/m3	[45, 55]
7	Specific heat capacity layer 2 (insulation)	1000 J/(kg·K)	[900, 1100]
8	Thickness layer 2 (insulation)	0.100 m	[0.09, 0.11]
9	Conductivity layer 3 (concrete)	1.200 W/(m·K)	[1.08, 1.32]
10	Density layer 3 (concrete)	2000 kg/m3	[1800, 2200]
11	Specific heat capacity layer 3 (concrete)	1000 J/(kg·K)	[900, 1100]
12	Thickness layer 3 (concrete)	0.200 m	[0.18, 0.22]
13	Internal convective heat transfer coefficient	2.5 W/(m2·K)	[2.25, 2.75]
14	External convective heat transfer coefficient	8 W/(m2·K)	[7.2, 8.8]
15	Solar absorptance coefficient of external wall surface	0.6	[0.54, 0.66]
16	Longwave emission coefficient of internal wall surface	0.9	[0.81, 0.99]
	Wall west
17	Conductivity Layer 1 (plaster)	0.140 W/(m·K)	[0.126, 0.154]
18	Density layer 1 (plaster)	800 kg/m3	[720, 880]
19	Specific heat capacity layer 1 (plaster)	1500 J/(kg·K)	[1350, 1650]
20	Thickness layer 1 (plaster)	0.005 m	[0.0045, 0.0055]
21	Conductivity layer 2 (insulation)	0.040 W/(m·K)	[0.036, 0.044]
22	Density layer 2 (insulation)	50 kg/m3	[45, 55]
23	Specific heat capacity layer 2 (insulation)	1000 J/(kg·K)	[900, 1100]
24	Thickness layer 2 (insulation)	0.100 m	[0.09, 0.11]
25	Conductivity layer 3 (concrete)	1.200 W/(m·K)	[1.08, 1.32]
26	Density layer 3 (concrete)	2000 kg/m3	[1800, 2200]
27	Specific heat capacity layer 3 (concrete)	1000 J/(kg·K)	[900, 1100]
28	Thickness layer 3 (concrete)	0.200 m	[0.18, 0.22]
29	Internal convective heat transfer coefficient	2.5 W/(m2·K)	[2.25, 2.75]
30	External convective heat transfer coefficient	8 W/(m2·K)	[7.2, 8.8]
31	Solar absorptance coefficient of external wall surface	0.6	[0.54, 0.66]
32	Longwave emission coefficient of internal wall surface	0.9	[0.81, 0.99]
	Wall north
33	Conductivity Layer 1 (plaster)	0.140 W/(m·K)	[0.126, 0.154]
34	Density layer 1 (plaster)	800 kg/m3	[720, 880]
35	Specific heat capacity layer 1 (plaster)	1500 J/(kg·K)	[1350, 1650]
36	Thickness layer 1 (plaster)	0.005 m	[0.0045, 0.0055]
37	Conductivity layer 2 (insulation)	0.040 W/(m·K)	[0.036, 0.044]
38	Density layer 2 (insulation)	50 kg/m3	[45, 55]
39	Specific heat capacity layer 2 (insulation)	1000 J/(kg·K)	[900, 1100]
40	Thickness layer 2 (insulation)	0.100 m	[0.09, 0.11]
41	Conductivity layer 3 (concrete)	1.200 W/(m·K)	[1.08, 1.32]
42	Density layer 3 (concrete)	2000 kg/m3	[1800, 2200]
43	Specific heat capacity layer 3 (concrete)	1000 J/(kg·K)	[900, 1100]
44	Thickness layer 3 (concrete)	0.200 m	[0.18, 0.22]
45	Internal convective heat transfer coefficient	2.5 W/(m2·K)	[2.25, 2.75]
46	External convective heat transfer coefficient	8 W/(m2·K)	[7.2, 8.8]
47	Solar absorptance coefficient of external wall surface	0.6	[0.54, 0.66]
48	Longwave emission coefficient of internal wall surface	0.9	[0.81, 0.99]
	Wall east
49	Conductivity Layer 1 (plaster)	0.140 W/(m·K)	[0.126, 0.154]
50	Density layer 1 (plaster)	800 kg/m3	[720, 880]
51	Specific heat capacity layer 1 (plaster)	1500 J/(kg·K)	[1350, 1650]
52	Thickness layer 1 (plaster)	0.005 m	[0.0045, 0.0055]
53	Conductivity layer 2 (insulation)	0.040 W/(m·K)	[0.036, 0.044]
54	Density layer 2 (insulation)	50 kg/m3	[45, 55]
55	Specific heat capacity layer 2 (insulation)	1000 J/(kg·K)	[900, 1100]
56	Thickness layer 2 (insulation)	0.100 m	[0.09, 0.11]
57	Conductivity layer 3 (concrete)	1.200 W/(m·K)	[1.08, 1.32]
58	Density layer 3 (concrete)	2000 kg/m3	[1800, 2200]
59	Specific heat capacity layer 3 (concrete)	1000 J/(kg·K)	[900, 1100]
60	Thickness layer 3 (concrete)	0.200 m	[0.18, 0.22]
61	Internal convective heat transfer coefficient	2.5 W/(m2·K)	[2.25, 2.75]
62	External convective heat transfer coefficient	8 W/(m2·K)	[7.2, 8.8]
63	Solar absorptance coefficient of external wall surface	0.6	[0.54, 0.66]
64	Longwave emission coefficient of internal wall surface	0.9	[0.81, 0.99]
	Ceiling
65	Conductivity Layer 1 (plaster)	0.140 W/(m·K)	[0.126, 0.154]
66	Density layer 1 (plaster)	800 kg/m3	[720, 880]
67	Specific heat capacity layer 1 (plaster)	1500 J/(kg·K)	[1350, 1650]
68	Thickness layer 1 (plaster)	0.005 m	[0.0045, 0.0055]
69	Conductivity layer 2 (insulation)	0.040 W/(m·K)	[0.036, 0.044]
70	Density layer 2 (insulation)	50 kg/m3	[45, 55]
71	Specific heat capacity layer 2 (insulation)	1000 J/(kg·K)	[900, 1100]
72	Thickness layer 2 (insulation)	0.100 m	[0.09, 0.11]
73	Conductivity layer 3 (concrete)	1.200 W/(m·K)	[1.08, 1.32]
74	Density layer 3 (concrete)	2000 kg/m3	[1800, 2200]
75	Specific heat capacity layer 3 (concrete)	1000 J/(kg·K)	[900, 1100]
76	Thickness layer 3 (concrete)	0.200 m	[0.18, 0.22]
77	Internal convective heat transfer coefficient	2.5 W/(m2·K)	[2.25, 2.75]
78	External convective heat transfer coefficient	8 W/(m2·K)	[7.2, 8.8]
79	Solar absorptance coefficient of external wall surface	0.6	[0.54, 0.66]
80	Longwave emission coefficient of internal wall surface	0.9	[0.81, 0.99]
	Floor
81	Conductivity Layer 1 (plaster)	0.140 W/(m·K)	[0.126, 0.154]
82	Density layer 1 (plaster)	800 kg/m3	[720, 880]
83	Specific heat capacity layer 1 (plaster)	1500 J/(kg·K)	[1350, 1650]
84	Thickness layer 1 (plaster)	0.005 m	[0.0045, 0.0055]
85	Conductivity layer 2 (insulation)	0.040 W/(m·K)	[0.036, 0.044]
86	Density layer 2 (insulation)	50 kg/m3	[45, 55]
87	Specific heat capacity layer 2 (insulation)	1000 J/(kg·K)	[900, 1100]
88	Thickness layer 2 (insulation)	0.100 m	[0.09, 0.11]
89	Conductivity layer 3 (concrete)	1.200 W/(m·K)	[1.08, 1.32]
90	Density layer 3 (concrete)	2000 kg/m3	[1800, 2200]
91	Specific heat capacity layer 3 (concrete)	1000 J/(kg·K)	[900, 1100]
92	Thickness layer 3 (concrete)	0.200 m	[0.18, 0.22]
93	Internal convective heat transfer coefficient	2.5 W/(m2·K)	[2.25, 2.75]
94	External convective heat transfer coefficient	8 W/(m2·K)	[7.2, 8.8]
95	Solar absorptance coefficient of external wall surface	0.6	[0.54, 0.66]
96	Longwave emission coefficient of internal wall surface	0.9	[0.81, 0.99]
	Window properties
97	Total surface area (window glazing + frame)	3 m2	[2.70, 3.30]
98	U-value (window glazing + frame)	1.10 W/(m2·K)	[0.99, 1.21]
99	g-value (SHGC—Solar Heat Gain Coefficient)	0.6	[0.54, 0.66]
	Operation parameters
100	Infiltration rate	0.5 1/h	[0.45, 0.55]
101	Convective internal heat gains	80 W	[72, 88]
102	Radiative internal heat gains	120 W	[108, 132]
103	Albedo (ground reflectance)	0.2	[0.18, 0.22]
	HVAC system
104	Heating set-point temperature	20 °C	[18, 22]
105	Cooling set-point temperature	28 °C	[25.2, 30.8]

(Appendix A) shows the list of input parameters included in the analysis and the assigned uncertainty ranges for each parameter. Due to the large number of input parameters involved in this study, an uncertainty of ±10% is considered for all parameters. It is worth mentioning that the proposed SA framework can also consider parameters with different uncertainty ranges. To build samples of input parameter sets, the factorial sampling plan proposed by Morris [73] is used. All input parameters are uniformly distributed, and it is assumed that there is no correlation between input parameters. Two statistical measures from Morris’ method are used to analyze the influence of each parameter on model outputs (annual heating, cooling, and total energy consumption): the mean of the absolute value μ* and the standard deviation σ of the elementary effects (see Section 2.2). A total of 44 uncertain parameters are selected from an original set of 105.

The main objective of the SA in this study is applying the Sobol’ method [74], which is one of the most robust and accurate variance-based SAs [55,75]. However, it requires a large number of samples and simulations, which incur considerable computational costs. Before conducting the Sobol’ method, meta-models are implemented in the second stage of the SA, which can be evaluated faster and reduce the computational cost of Sobol’ indices in comparison with the direct simulation of the original BEPS model. To generate a dataset for developing meta-models, Latin Hypercube sampling (LHS) in combination with Monte Carlo simulation is applied in this study.

LHS is a particular form of stratified sampling method and is commonly used for uncertainty and sensitivity analysis in computationally demanding models. Due to its conceptual simplicity and efficient stratification over the range of each uncertain parameter, LHS is very popular [76]. The samples (N = 400) for each of the selected uncertain parameters (k = 44) are created using their probability distributions and stored in a matrix X as:

$$X=\left[\begin{array}{cccc}{x}_{\mathrm{1,1}}& x_{\mathrm{2,1}}& \cdots & x_{k,1}\\ x_{\mathrm{1,2}}& x_{\mathrm{2,2}}& \cdots & x_{k,2}\\ ⋮& ⋮& \ddots & ⋮\\ x_{1,N}& x_{2,N}& \cdots & x_{k,N}\end{array}\right]$$

(1)

After generating the sample matrix, 400 Monte Carlo simulations (one simulation per row of the sample matrix) are carried out in Dymola (deterministic simulation) using a Functional Mock-up Interface (FMI) for co-simulation [77] to calculate the simulation output Y for each sample as:

$$Y=\left[\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_N\end{array}\right]=\left[\begin{array}{cccc}f(x_{\mathrm{1,1}},& x_{\mathrm{2,1}},& \cdots ,& x_{k,1})\\ f(x_{\mathrm{1,2}},& x_{\mathrm{2,2}},& \cdots ,& x_{k,2})\\ ⋮& ⋮& \ddots & ⋮\\ f(x_{1,N},& x_{2,N},& \cdots ,& x_{k,N})\end{array}\right]$$

(2)

The FMI provides a definition of an interface standard to facilitate co-simulation and integrate the models into different modeling environments. The models can be encapsulated into a Functional Mock-up Unit (FMU) and used in other simulation tools.

These simulation runs constitute the database, which is randomly partitioned into two sets for training and testing meta-models. The training set (70% of simulation runs) is used to develop meta-models. The remaining 30% of simulation runs are used as a testing set to assess the accuracy and reliability of the developed meta-models.

Four commonly used meta-models, i.e., MARS, PR, RF, and SVR based on the RBF kernel, are assessed. The performances of the developed meta-models are compared by calculating six metrics: MBE, NMBE, RMSE, CVRMSE, R², and GOF (see Section 2.1).

In the third stage of the SA, the variance-based SA by Sobol’ is applied to impart more quantitative information concerning parameter significance. In the Sobol’ method, the contribution of each uncertain parameter to the total variance of the output is evaluated by first-order ($S_i$) and total-order ($S_{Ti}$) Sobol’ indices using the Sobol’ sequence sampling (see Section 2.4). Approximately 23,040 simulation runs are carried out using the MARS model as a meta-model of the original complex Dymola model. The most sensitive parameters to the simulation model output (annual total energy consumption) are identified. The uncertainty and sensitivity analysis approach used in the second part of the methodology is shown in Figure 2.

2.1. Statistical Performance Metrics

Six statistical metrics are used to verify and quantify the accuracy of the model.

Table 2. Statistical performance metrics.

Metric	Formula	Equation
MBE	${\displaystyle \frac{{\sum }_{k=1}^n(r-p)}{n}}$	(3)
NMBE	${\displaystyle \frac{{\sum }_{k=1}^n\left(r-p\right)}{n\times \overline{r}}}\times 100$	(4)
RMSE	$\sqrt{{\displaystyle \frac{{\sum }_{k=1}^n{(r-p)}^2}{n}}}$	(5)
CVRMSE	${\displaystyle \frac{\sqrt{{\displaystyle \frac{{\sum }_{k=1}^n{(r-p)}^2}{n}}}}{\overline{r}}}\times 100$	(6)
GOF	${\displaystyle \frac{\sqrt{2}}{2}}\times \sqrt{{\mathrm{C}\mathrm{V}\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}}^2+{\mathrm{N}\mathrm{M}\mathrm{B}\mathrm{E}}^2}$	(7)
R2	$1-{\displaystyle \frac{{\sum }_{k=1}^n{(r-p)}^2}{{\sum }_{k=1}^n{(r-\overline{r})}^2}}$	(8)

presents the list of metrics used in this study.

MBE indicates the average of the model errors and is calculated using Equation (3), where r is the reference (or original) value, p is the predicted value, and n is the total number of data points. A value of MBE close to zero is desirable. NMBE is calculated by dividing the MBE by the mean of the reference values ($\overline{r}$) and is defined according to Equation (4). RMSE indicates the square root of the average of the squared model errors and is introduced in Equation (5). CVRMSE is a normalized form of RMSE and is calculated by dividing the RMSE by the mean of the reference values ($\overline{r}$) and is defined according to Equation (6).

The CVRMSE and the NMBE are dimensionless metrics. ASHRAE Guideline 14 [78], Measurement and Verification of the Federal Energy Management Program (FEMP) [79], and International Performance Measurement and Verification Protocol (IPMVP^®) [80] prescribe two metrics, NMBE and CVRMSE, for the evaluation of the model’s accuracy. The Goodness of Fit (GOF) proposed by ASHRAE 1051-RP [81] combines the two normalized metrics NMBE and CV (RMSE) into a single metric, which is calculated according to Equation (7). The R² indicates how well prediction values fit reference values and is described according to Equation (8). The value of R² ranges between 0 and 1. The closer R² is to 1, the greater the accuracy of the prediction model.

2.2. Morris’ Method

A generalized model is given by a mathematical function according to Equation (9):

$$Y=f\left(X\right)$$

(9)

where X is a k × N matrix with N samples of k input parameters under consideration (see Equation (1)), and Y is a vector of the corresponding model output (see Equation (2)).

To detect the most important input parameters from among a large number, the screening-based SA proposed by Morris in combination with a factorial sampling plan is implemented. The aim is to determine the subset of parameters that can be fixed at their baseline point values without considerably decreasing the output variance. Morris’ method is generally used to discover which input parameters may be considered as (a) negligible or unimportant, (b) linear and additive, or (c) non-linear or involved in interactions with other parameters [82,83]. The considerable advantage of the screening-based SA by Morris is that the assessment of the influence of input parameters can be carried out with a small sample size. In cases of large numbers of parameters, Morris’ method is a felicitous screening-based technique prior to performing variance-based SA to alleviate the computational cost [83].

The samples are representative of the parameter space Ω, which is discretized by dividing the range of each parameter into a number of p levels with a predetermined distance of equal size ∆. Each input parameter varies once with step ∆ that builds t trajectories of (k + 1) points in Ω. Evaluation of the influence of each parameter on the model output is carried out by calculating a so-called elementary effect (EE), according to Equation (10).

$${EE}_i=\frac{f\left(x_1,\cdots ,x_{i-1},x_i\pm ∆,\cdots ,x_k\right)-f(x_1,\cdots ,x_k)}{∆}$$

(10)

The computation of each elementary effect requires two sample points. Therefore, the number of required simulation runs (N) is given by N = t × (k + 1), where t is the number of elementary effects per input parameter and k is the number of input parameters. The values between 4 and 10 are recommended for t [34,73]. Due to the large number of parameters in this study (k = 105), the number of trajectories t = 4 is used. Therefore, 4 × (105 + 1) = 424 simulation runs are required to perform the screening-based SA by Morris.

The sensitivity of the model output to each input parameter is analyzed by two statistical measures: the mean value (μ) and the standard deviation (σ) of the elementary effects, as defined in Equations (11) and (12):

$${\mu }_i=\frac{1}{t}\sum _{j=1}^t{EE}_i^j$$

(11)

$${\sigma }_i=\sqrt{\frac{1}{t-1}\sum _{j=1}^t{\left({EE}_i^j-{\mu }_i\right)}^2}$$

(12)

where ${EE}_i^j$ represents the elementary effects of the input parameter i computed along trajectory j (${EE}_i^j$, i = 1, 2, …, k and j = 1, 2, …, t). The mean value of elementary effects (μ) indicates a measure of the overall effect (sensitivity) of an input parameter, while the standard deviation of elementary effects (σ) indicates a measure of interactions of an input parameter with other parameters, i.e., non-linear effects of an input parameter. These two sensitivity measures are basically applied as indicators of which parameters can be considered important [73]. Campolongo et al. [82] proposed using the mean of the absolute value of elementary effects (μ*), as presented in Equation (13), since positive and negative values of elementary effects can cancel each other out (especially where input parameters are involved in interaction effects).

$${\mu }_i^{*}=\frac{1}{t}\sum _{j=1}^t\left|{EE}_i^j\right|$$

(13)

The μ* is used as an indicator to rank parameters in order of importance.

2.3. Sobol’ Method

Sobol’ method is one of the variance-based GSAs proposed by Sobol’ [74]. It is based on the decomposition of the total model output variance and can consider the main effects as well as the interaction effects of parameters (uncertainty in single parameters as well as uncertainty in combinations of parameters). The key aim of this method is that the contribution of each uncertain parameter to the total variance of the model output is quantified by specific order sensitivity indices, including first-order Sobol’ index ($S_i$), second-order Sobol’ index ($S_{ij}$), …, and total-order Sobol’ index ($S_{Ti}$).

The variance of a model output V(Y) can be defined as a sum of the variances of first-order and higher-order functions:

$$V\left(Y\right)=\sum _{j=1}^k{V}_i+\sum _{j=1}^k\sum _{i<j}^k{V}_{i,j}+\dots +V_{\mathrm{1,2},\dots ,k}$$

(14)

where $V_i$ represents the variance contribution of first-order interaction effects for the input parameter i and $V_{i,j}$ indicates the variance contribution of second-order interaction effects between two parameters.

The first-order Sobol’ sensitivity index ($S_i$) is written as follows:

$$S_i=\frac{V_{X_i}\left(E_{X_{~i}}\left(Y|X_i\right)\right)}{V\left(Y\right)}$$

(15)

where $E_{X_{~i}}\left(Y|X_i\right)$ represents the mean of the model output Y taken over all values of the sample matrix of input parameters X when the parameter $X_i$ remains fixed and all other input parameters are changed within the defined range. The term $X_{~i}$ indicates a matrix of all input parameters, but $X_i$. The first-order index $S_i$ is applied to determine the most influential parameters on the variance of the model output. However, it does not describe the interactions of an input parameter with other parameters. Thus, a low $S_i$ does not express that the parameter $X_i$ is noninfluential, since there can be higher order interaction effects.

The total-order Sobol’ sensitivity index ($S_{Ti}$) considers all interaction effects of the input parameter $X_i$ with other parameters (first and all higher order interaction effects) and can be defined as:

$$S_{Ti}=\frac{E_{X_{~i}}\left(V_{X_i}\left(Y|X_{~i}\right)\right)}{V\left(Y\right)}$$

(16)

The total-order index ($S_{Ti}$) is used to determine unimportant (negligible) parameters, i.e., parameters without significantly affecting the variance of the model output.

The sampling from parameter distributions is designed using the Sobol’ sequence, which is a quasi-random low-discrepancy sampling technique, to generate uniform samples of the parameter space Ω. Carrying out variance-based SA by Sobol’ method for a model with k input parameters entails a computational cost of N × (2k + 2) simulation runs, where N is the sample size used for the Monte Carlo simulation [49]. In this study, to perform the robust SA, 23,040 simulation runs are executed using the MARS meta-model.

2.4. Meta-Modeling

The PR is an extension of the linear model that accommodates non-linear relationships between input parameters and simulation outputs. Since polynomial functions of parameters are included in the regression model, it is known as polynomial regression. In this study, the second-order polynomial model is applied, which can be described in Equation (17):

$$\widehat{Y}={\beta }_0+\sum _{i=1}^k{\beta }_i{x}_i+\sum _{i=1}^k{\beta }_{ii}{x}_i^2+\sum _{i=1}^k\sum _{j=1,i<j}^k{\beta }_{ij}{x}_i{x}_j$$

(17)

where $\widehat{Y}$ is the predicted value, x is the input parameter, and β is the regression coefficient. For k parameters, the second-order polynomial model of Equation (17) will have p = (k + 1) × (k + 2)/2 terms [84].

The RF is a tree-based machine learning algorithm that applies trees as building blocks to construct prediction models [85]. The RF model breaks down the dataset into tree-structure forms and uses averaging to estimate the output of all trees. In this study, the number of trees in the forest was set at 600.

The Support Vector Machine (SVM) is an artificial intelligence (AI) algorithm developed by Vapnik [86]. The goal of the SVM is to minimize the generalization error bound and, furthermore, to obtain generalized performance. That is, the SVM attempts to minimize the deviation (ε) of the prediction function from the target. In this study, the RBF is used as a kernel function [87].

The MARS [88] is a nonparametric, automated regression-based machine learning algorithm that uses labeled samples to model the complex non-linear relationship between input parameters (independent variables) and simulation outputs (dependent variables) and, subsequently, to predict outputs (targets). It can be considered a generalization of stepwise linear regression and has an outstanding ability to tackle high-dimensional problems with a large number of input parameters [62,68,69]. The MARS algorithm can be used to identify the parameters that have a considerable influence on simulation outputs, which is essential in the early stages of the building design process.

MARS uses spline basis functions of the form ${\left(x_v-t\right)}_{+}$ and ${\left(t-x_v\right)}_{+}$, defined according to Equations (18) and (19).

$${\left(x_v-t\right)}_{+}=\left\{\begin{array}{l}{x}_v-tif{x}_v>t\\ 0otherwise\end{array}\right.$$

(18)

$${\left(t-x_v\right)}_{+}=\left\{\begin{array}{c}t-x_{v}if{x}_v<t\\ 0otherwise\end{array}\right.$$

(19)

where $x_v$ is the v-th independent variable (input parameter), t is the knot of the spline, and the subscript “+” means (.)₊ takes the positive part of the inner argument. Each function is piecewise linear, with a knot at the value t. Figure 3 presents the spline basis functions (t = 0.5).

The general expression of the MARS model can be described according to Equation (20):

$$\widehat{Y}=\widehat{f}\left(X\right)={\beta }_0+\sum _{m=1}^M{\beta }_m{h}_m\left(X\right)$$

(20)

where $\widehat{Y}$ is the dependent variable (output) predicted by the MARS model by means of a function $\widehat{f}\left(X\right)$, ${\beta }_0$ is an initial constant, ${\beta }_m$ is the coefficient of m-th basis function, $h_m\left(X\right)$ is the m-th basis function, which can be a single spline basis function or a product of two or more such functions, and M is the number of basis functions, h_m. The basis functions $h_m\left(X\right)$ take the form as follows:

$$h_m\left(X\right)=\prod _{k=1}^{K_M}\left[s_{k,m}(x_{v(k,m)}-t_{k,m})\right]$$

(21)

where $K_M$ is the number of knots (number of interaction order in m-th basis function), $s_{k,m}$ has the value of either +1 or −1, which determines the right/left sense of the associated step function, and $t_{k,m}$ indicates the knot location on each of the corresponding variables.

3. Case Study

A hypothetical building with one single thermal zone (shoebox-shaped building) and one singular south-facing window without shading is considered an initial case study. The building has dimensions of 2 m in width, 5 m in length, and a floor-to-ceiling height of 3 m, totaling its interior volume at 30 m³. The window is centered in the southern wall and has a total surface area of 3 m² (1.5 m × 2 m), as shown in Figure 4. All surfaces are modeled as exterior walls. The material properties of the building envelope used in this study can be found in

Table 3. Material properties of the building envelope.

Layer Nr.	Components (Outside to Inside)	Thickness d [m]	Thermal Conductivity $\mathsf{\lambda }\left[\frac{\mathbf{W}}{\mathbf{m}·\mathbf{K}}\right]$	Density $\mathsf{\rho }\left[\frac{\mathbf{k}\mathbf{g}}{{\mathbf{m}}^3}\right]$	Specific Heat Capacity $\mathit{C}\left[\frac{\mathbf{J}}{\mathbf{k}\mathbf{g}·\mathbf{K}}\right]$
1	Plaster	0.005	0.14	800	1500
2	Thermal insulation	0.100	0.04	50	1000
3	Concrete	0.200	1.20	2000	1000

. The same boundary conditions are applied to all surfaces, and an initial temperature of 20 °C is defined for the interior air volume and wall instances. The infiltration rate of 0.5 air changes per hour is assumed for the building air volume. No heat recovery is considered. The internal heat gains are set to 200 W. The radiative and convective portions of internal heat gains are taken as 60% and 40%, respectively.

The building is assumed to be heated and cooled by an ideal HVAC system. The maximum heating power of the HVAC system is limited to 1000 W, while the maximum cooling power is capped at 1200 W. The heating set-point temperature is 20 °C, and the cooling set-point temperature is set to 28 °C. The proposed case study is modeled in Dymola using the Modelica^® language (Version 3.2.2) [89] and simulated for an entire year using a time step resolution of one hour. The investigated building model is controlled based on the heating/cooling set-point temperature.

For this analysis, the simulations are carried out using the weather data of Denver-Stapleton (Denver, CO, USA), which is a cool-dry climate (zone 5B) according to the ANSI/ASHRAE/IES Standard 90.1 [90]. The original weather data were provided in the EnergyPlus Weather (*.epw) format, which is converted to the Modelica timeseries script (*.mos) before running the simulation in Dymola. The climate data consists of hourly values of solar radiation and meteorological conditions for a 1-year period. Figure 5 shows the outdoor dry bulb temperature and the global horizontal irradiance (GHI) for one year.

4. Results and Discussion

This section presents the results of the application of the proposed framework with respect to the investigated case study of a building with one single thermal zone.

4.1. Validation of Deterministic Simulations

The scope of this section is to assess the results of two deterministic simulations (reference and synthetic), i.e., to reveal the inconsistencies between the two simulation results. In the reference simulation, the aforementioned case study was used to carry out the simulation. In the synthetic simulation, the virtual building model, which has different input parameters related to the thermophysical and optical properties of the building envelope, infiltration rate, internal heat gains, and HVAC system, was simulated.

Both deterministic simulations are performed for one year at an hourly resolution using the same simulation setup.

Figure 6 compares the results of heating and cooling loads. It can be qualitatively observed that there are discrepancies between reference and synthetic models for both heating and cooling load predictions. There are no heating loads from March to November. The maximum heating load predictions have values of 581.7 W and 534.5 W (in January) for the reference model and synthetic model, respectively. The cooling load predictions have maximum values of 906.8 W and 844.9 W (in July) for the synthetic model and reference model, respectively.

To quantify and ascertain the discrepancies between reference and synthetic models, six statistical metrics, described in Section 2.1, were used. The results of the investigated performance metrics for the cooling load are shown in

Table 4. Performance evaluation for cooling load.

Metrics	Comparison of Reference and Synthetic Models
MBE [W]	−32.28
NMBE [%]	−15.18
RMSE [W]	43.10
CVRMSE [%]	20.27
GOF [%]	17.91
R2	0.97

. It indicates deviations in reference and synthetic simulation results. To discover the most important model parameters that contribute enormously to these discrepancies, a meta-model-based SA was conducted.

4.2. Sensitivity Analysis Using Morris’ Method

The screening-based SA using Morris’ method was performed to determine the most influential input parameters on annual heating, cooling, and total consumption. A total of 105 uncertain parameters, including thermophysical and optical properties, infiltration rate, internal heat gains, and HVAC system, were selected. The factorial sampling plan proposed by Morris was applied to build 424 samples of the input parameter.

Figure 7 illustrates the results of Morris’ sensitivity analysis of building energy simulations for the investigated case study.

Each point in scatter plots indicates the impact of each input parameter included in the investigation on the selected simulation output. The horizontal axis represents the mean of the absolute value of the elementary effects (μ*), which indicates the overall importance of an input parameter. The vertical axis represents the standard deviation of the elementary effects (σ), which describes the total interactions of an input parameter with other parameters involved in the analysis (non-linear effects of parameters). It is also worth pointing out that according to Equation (11) the elementary effect has the same unit as the simulation output, which is the annual energy consumption (kWh/year) in this study. The ratio σ/μ* is used as an indicator for the assessment of the linearity (and monotonicity) of parameters. The ratio σ/μ* = 0 is considered for the linear behavior of parameters. Three slope lines (σ/μ* = 0.1, 0.5, and 1) can be plotted in the scatter plot (Figure 7) to classify parameters into four zones: almost linear (below σ/μ* = 0.1), monotonic (0.1 < σ/μ* < 0.5), almost monotonic (0.5 < σ/μ* < 1.0), and non-linear/non-monotonic (σ/μ* > 1) [54].

The mean of the absolute value of the elementary effects (μ*) is used for ranking the input parameters in accordance with their importance. The higher the μ* value, the more important (and sensitive) the parameter. Figure 8 shows the ranking of 44 input parameters from an initial set of 105 for the simulation output of annual total consumption.

Table 5. Selected input parameters and their μ* by Morris’ method.

Rank	Parameter	Unit	Index	μ*
1	Cooling set-point	[°C]	105	841.97
2	g-value window	-	99	300.56
3	Window area	m2	97	220.35
4	Heating set-point	[°C]	104	129.51
5	Internal heat gain (radiative)	W	102	112.71
6	Albedo	-	103	80.10
7	Internal heat gain (convective)	W	101	77.70
8	Solar absorptance ceiling	-	79	58.59
9	External convective coefficient wall west	W/(m2·K)	30	57.46
10	Solar absorptance west	-	31	56.35
11	External convective coefficient ceiling	W/(m2·K)	78	54.48
12	External convective coefficient wall east	W/(m2·K)	62	47.37
13	Solar absorptance east	-	63	46.00
14	Infiltration rate	1/h (ACH)	100	40.04
15	Thickness floor layer 2	m	88	27.09
16	U-value window	W/(m2·K)	98	25.39
17	Thickness ceiling layer 2	m	72	22.43
18	Conductivity ceiling layer 2	W/(m·K)	69	20.78
19	Conductivity floor layer 2	W/(m·K)	85	18.80
20	Conductivity wall east layer 2	W/(m·K)	53	16.28
21	External convective coefficient floor	W/(m2·K)	94	14.77
22	Internal convective coefficient wall east	W/(m2·K)	61	14.35
22	External convective coefficient wall south	W/(m2·K)	14	14.35
24	Internal convective coefficient wall west	W/(m2·K)	29	14.25
25	Solar absorptance wall south	-	15	14.04
26	Thickness wall west layer 2	M	24	13.58
27	Conductivity wall west layer 2	W/(m·K)	21	13.05
28	Thickness wall north layer 2	M	40	11.65
29	Solar absorptance floor	-	95	11.50
30	External convective coefficient wall north	W/(m2·K)	46	11.21
31	Conductivity wall north layer 2	W/(m·K)	37	9.94
32	Solar absorptance wall north	-	47	9.72
33	Internal convective coefficient ceiling	W/(m2·K)	77	9.03
34	Internal convective coefficient floor	W/(m2·K)	93	7.48
35	Thickness wall east layer 2	m	56	6.27
36	Internal convective coefficient wall north	W/(m2·K)	45	5.54
37	Specific heat wall west layer 3	J/(kg·K)	27	4.12
38	Specific heat wall east layer 3	J/(kg·K)	59	4.03
39	Density wall east layer 3	kg/m3	58	3.72
40	Density wall west layer 3	kg/m3	26	3.71
41	Thickness wall south layer 2	m	8	3.52
42	Conductivity wall south layer 2	W/(m·K)	5	3.30
43	Internal convective coefficient wall south	W/(m2·K)	13	2.74
44	Thickness ceiling layer 3	m	76	2.55

shows the selected input parameters as well as their rank with the corresponding μ* and parameter index. In this analysis, cooling set-point temperature was identified as the most influential parameter with the highest value of μ*, having a monotonic effect on the annual total consumption, followed by the g-value of the window, window area, and heating set-point temperature.

4.3. Meta-Modeling

In this section, four meta-models (MARS, PR, RF, and SVR) based on the RBF kernel were developed to evaluate their performance. LHS, in combination with Monte Carlo simulation, was used to generate a dataset for developing meta-models for annual total consumption.

Trading off the accuracy and the computational cost, 400 samples for each of the 44 uncertain input parameters (see Table 5) were created using uniform probability distributions. It is assumed that there is no correlation between input parameters. These simulation runs constitute the database, which is randomly partitioned into two sets for training (70% of simulation runs) and testing (30% of simulation runs) meta-models.

Six statistical metrics were applied to compare the performance of the developed meta-models.

Table 6. Performance evaluation of meta-models.

Meta-Model	MBE [kWh]	NMBE [%]	RMSE [kWh]	CVRMSE [%]	GOF [%]	R2
MARS	−0.73	−0.04	33.60	1.74	1.23	0.99
PR	−13.32	−0.69	52.48	2.71	1.98	0.97
RF	−7.94	−0.41	104.32	5.39	3.82	0.88
SVR (kernel: RBF)	−9.28	−0.48	115.56	5.97	4.24	0.85

shows that MARS considerably outperforms all other meta-models with respect to annual total consumption predictions.

The R² values for MARS and PR are very close together at 0.99 and 0.97, respectively. However, the MBE value for MARS (−0.73 kWh) differs significantly from that for the PR model (−13.32 kWh). It can be interpreted that there may be a bias (i.e., systematic error) in the predicted values of the PR model. The negative value of MBE describes that the average of predicted values of the meta-model during the time period is higher than the average of Dymola values, which indicates the overprediction of the meta-model. Figure 9 graphically visualizes the values of CVRMSE, GOF, and R² for each meta-model examined, which confirms the superior performance of the MARS algorithm.

Figure 10 shows a scatter plot created between the MARS meta-model and the Dymola model for the whole generated sample. The x-axis represents Dymola simulation results, and the y-axis represents MARS predictions. The red line indicates the perfect fit line (y = x). It can be shown that the investigated MARS model provides an accurate prediction for the annual total consumption and is in very good agreement. The MARS is considered a meta-model of the original Dymola model for further SA, which is described in the next section.

4.4. Sensitivity Analysis Using Sobol’ Method

The Saltelli’s extension of the Sobol’ sequence was used to generate uniform samples of parameter space Ω. With a sample size of N = 256 and k = 44 uncertain input parameters (see Table 5), 23,040 evaluation runs (N × (2k + 2)) were carried out using the MARS meta-model to perform the variance-based SA using the Sobol’ method. The most sensitive parameters to the model output (annual total consumption) are shown in Figure 11 as total effects ST from the Sobol’ method. The total sensitivity index ST indicates the total effects (first-order and all parameter interaction effects) of parameters on model output and can be considered a measure of negligible input parameters.

The ranking of the top 16 most influential input parameters (from an initial set of 105) with the corresponding total sensitivity index ST is given in

Table 7. Input parameter rankings and their total sensitivity index by Sobol’ method.

Rank	Parameter	Unit	ST
1	Cooling set-point	[°C]	0.7656
2	g-value window	-	0.0960
3	Window area	m2	0.0768
4	Heating set-point	[°C]	0.0140
5	Internal heat gain (radiative)	W	0.0113
6	Albedo	-	0.0067
7	Internal heat gain (convective)	W	0.0063
8	Solar absorptance ceiling	-	0.0040
9	External convective coeff. ceiling	W/(m2 K)	0.0034
10	Solar absorptance west	-	0.0030
11	External convective coeff. east	W/(m2 K)	0.0025
12	Solar absorptance east	-	0.0021
13	External convective coeff. west	W/(m2 K)	0.0021
14	Infiltration rate	1/h (ACH)	0.0020
15	U-value window	W/(m2 K)	0.0016
16	Solar absorptance south	-	0.0012

. The remaining parameters exhibit very low ST values, i.e., they have no significant impact on the total variance of the model output.

It is observed that the first eight input parameters, i.e., cooling set-point temperature, g-value of the window, window area, heating set-point temperature, radiative portion of the internal heat gain, albedo, convective portion of the internal heat gain, and solar absorptance of the ceiling, are consistent with the SA from Morris’ method (Table 5).

Nevertheless, there is considerable disagreement concerning the solar absorptance of the southern wall, which is ranked higher than by Morris’ method. It can be inferred that the solar absorptance of the southern wall is involved in interaction effects with other input parameters or non-linear behavior.

5. Conclusions and Future Work

5.1. Conclusions

BEPS is a function of many input parameters. Due to large uncertainties and the complex non-linear behavior of parameters, a considerable mismatch can occur between BEPS output and reference data. Furthermore, carrying out stochastic simulations with a vast number of samples in BEPS tools is very time-consuming. In this paper, a novel four-stage SA framework based on the MARS meta-model is proposed to explore significant uncertain input parameters that the BEPS output (annual energy consumption) is most sensitive to and to provide an overview of critical parameters in a holistic context. The major contribution of this study can be formulated as follows:

• In the first stage, two deterministic simulations are carried out. The comparative testing validation method is used to assess the deviation of the simulation results.
• In the second stage, the screening-based SA using Morris’ method is performed to identify the most important input parameters, which have a significant influence on the model output (annual energy consumption). In this stage, the number of uncertain parameters in the investigated case study is reduced from 105 to 44 prior to performing variance-based SA. The selected input parameters are used for constructing the meta-model.
• In the third stage, meta-modeling techniques are applied to tackle the computational costs of the variance-based SA, especially in cases of large numbers of time-consuming simulation runs. LHS, in combination with Monte Carlo simulation, is used to propagate uncertainties by running 400 simulation runs in the Dymola model and to generate a database for training and testing meta-models. A comparison study of the four meta-models, MARS, PR, RF, and SVR (based on the RBF kernel), is conducted. The results show that MARS is the most accurate meta-model in this study and provides a very precise prediction for the annual energy consumption compared to the original Dymola model.
• Finally, in the fourth stage, the variance-based SA using Sobol’ method is performed as a final, robust stage of SA. MARS is applied to implement uncertainty propagation based on Monte Carlo simulations (23,040 simulation runs). Running the MARS meta-model can be implemented faster and reduce the computational cost of Sobol’ sensitivity indices in comparison with the direct simulation of complex building models.

Due to using meta-modeling techniques and implementing two GSAs (Morris and Sobol’ methods), the proposed framework provides the fast, reliable SA of highly sophisticated building models, which is required in the early building design process for robust decision-making. In addition, this framework considers a large set of uncertain parameters, which increases the reliability of uncertainty and sensitivity analysis.

It can be concluded that the cooling set-point temperature has a strong influence on the annual total consumption with the total-order Sobol’ index of $S_T$ = 0.76 for the examined case study. Other highly influential parameters are those related to window parameters, i.e., the g-value and area of the window. In addition to this, this framework highlights that performing the screening-based SA (Morris’ method) and MARS meta-model can reduce the computational cost of Sobol’ sensitivity indices dramatically in comparison with the direct simulation of the original, complex building model. Therefore, the interpretation and evaluation of model outputs can be made faster and more robust.

5.2. Future Work

The proposed framework is based on weather data, which represents a cool-dry climate. The SA results could significantly differ if other weather data (such as hot-dry climate) is used. Further research work is required to improve the framework concerning several weather data points. Furthermore, MARS, in combination with Sobol’ method, is used in this study to explore the impacts of uncertain input parameters on annual energy consumption. In future work, other meta-models in combination with different variance-based SAs (such as FAST) will be investigated to compare with the developed SA framework. However, machine learning algorithms are often uninterpretable black-box models and lack transparency. Indeed, for these algorithms, it is extremely hard to explain how predictions were made. Further research is required to apply explainable artificial intelligence (XAI) approaches, such as Local Interpretable Model-Agnostic Explanations (LIME) [91] and Shapley Additive exPlanation (SHAP) [92], to improve the interpretability of machine learning-based systems.