Validation of Calibrated Energy Models: Common Errors

Abstract

Nowadays, there is growing interest in all the smart technologies that provide us with information and knowledge about the human environment. In the energy field, thanks to the amount of data received from smart meters and devices and the progress made in both energy software and computers, the quality of energy models is gradually improving and, hence, also the suitability of Energy Conservation Measures (ECMs). For this reason, the measurement of the accuracy of building energy models is an important task, because once the model is validated through a calibration procedure, it can be used, for example, to apply and study different strategies to reduce its energy consumption in maintaining human comfort. There are several agencies that have developed guidelines and methodologies to establish a measure of the accuracy of these models, and the most widely recognized are: ASHRAE Guideline 14-2014, the International Performance Measurement and Verification Protocol (IPMVP) and the Federal Energy Management Program (FEMP). This article intends to shed light on these validation measurements (uncertainty indices) by focusing on the typical mistakes made, as these errors could produce a false belief that the models used are calibrated.

1. Introduction

One of the main focuses of governments’ policies in almost every country is the energy supply. Energy security has a direct impact on social welfare, the level of the economy and the safety of a country, and for this reason, all possible energy savings in all sectors are important. The International Energy Agency highlights that the buildings sector is responsible for over one-third of final energy consumption, and therefore, it is an important source of CO${}_2$ emissions [1]. Fortunately, the energy savings potential of the buildings sector is also high (approximately 10% and 40% in hot climates), and for this reason, the optimization of its energy consumption is a key factor in the governments’ road map.

In this scenario, the Measurement and Verification (M&V) protocols are critical due to their capacity to evaluate the scope of each Energy Conservation Measure (ECM). Therefore, trustworthy energy saving calculations are crucial to encourage Energy Services Companies (ESCOs), building owners, utilities, etc., to make investments in energy projects to improve building energy efficiency and reduce CO${}_2$ emissions.

The relentless advance of technology (the rising availability of “smart meters” cheaper, more accurate and better simulation software, powerful computers, etc.) is increasing this “trust” in energy saving calculations where energy models play a key role [2,3,4]. Nowadays, the whole-building approach of Building Energy Models (BEMs) is more common than the single measure approaches due to the increase levels of automation, lower costs, etc. [5,6,7]. The use of these energy models is not only to study ECMs, but also to use them as a Model Predictive Control (MPC), as software-as-a-service to track the energy consumption, or as a tool to study risks in building energy management for insurance companies, etc. The accuracy of the BEM used is a determining factor for all these applications; therefore, the use of calibrated models is required.

ASHRAE guidelines 14-2014 defines calibration as the: “(b) process of reducing the uncertainty of a model by comparing the predicted output of the model under a specific set of conditions to the actual measured data for the same set of conditions. (…) calibration includes following defined procedures that identify what parameters of the instrument, meter, or model may be adjusted, determining what is an acceptable level of accuracy or uncertainty, and documenting the process and results [8]”. As can be seen, a calibrated model is one that, under the same set of conditions, can reproduce the measured data; and its accuracy is measured by an uncertainty analysis.

This article analyzes the most common uncertainty indices used in the process of calibration (Section 2), but focuses mainly on both the typical mistakes made when using these indices and their magnitude (Section 3). Section 4 explains the origin of a common error in the use of these uncertainty indices and Section 5 its spread through different documents (journals, thesis, reports, etc.). Section 6 shows the conclusions. Finally, we wish to stress that we hope not to offend anyone cited in this paper because the mistakes highlighted are typical due to the unclear existing references.

2. Measuring Uncertainty

As ASHRAE Guideline 14-2002 and 14-2014 explain, the uncertainty analysis is the “(b) process of determining the degree of confidence in the true value when using measurement procedures and/or calculations” [8,9]. The three main sources that explain how to determine this “degree” of confidence, its uncertainty, are ASHRAE Guideline 14, FEMP [10,11,12] and IPVMP [13,14,15,16]. They use simplified methods to quantify the uncertainty in saving computations and, as ASHRAE Guideline 14 explains, others can be used if they are compliant with the Guideline (Subsection 5.2.11 [9] or 4.2.11 [8]).

The principal uncertainty indices used are: Normalized Mean Bias Error ($NMBE$), Coefficient of Variation of the Root Mean Square Error ($CV\left(RMSE\right)$) and coefficient of determination ($R^2$). However, we start the explanation of these indices by defining the Mean Bias Error ($MBE$), due to its importance when the analysis and magnitude of the error is explained (Section 3).

$MBE$ (Mean Bias Error), as its name indicates, is the average of the errors of a sample space. Generally, it is a good indicator of the overall behavior of the simulated data with regards to the regression line of the sample. In Equation (1), $m_i$ is the measured value, $s_i$ is the simulated one and n the number of measured data points. Positive values mean that the model under-predicts measured data, and a negative one means over-prediction. However, the main problem with this index is that it is subject to cancellation errors where the sum of positive and negative values could reduce the value of $MBE$. In fact, when a model is calibrated or near calibrated, the regression line of the sample is so close to the simulated one that the cancellation effect increases considerably.

$$MBE=\frac{{\sum }_{i=1}^n(m_i-s_i)}{n}$$

(1)

$NMBE$ (Normalized Mean Bias Error) is a normalization of the $MBE$ index that is used to scale the results of $MBE$, making them comparable. It quantifies the $MBE$ index by dividing it by the mean of measured values ($\overline{m}$), giving the global difference between the real values and the predicted ones. In Equation (2), p is the number of adjustable model parameters, which, for calibration purposes, is suggested to be zero [17,18]. As in the case of $MBE$, positive and negative values mean the under- or over-prediction of this normalization. ASHRAE Guidelines [8,9] subtract measured values ($m_i$) from simulated ones ($s_i$) instead of FEMP [11,12] and IPMVP [16], which do the opposite. For this reason, the explanation of the under- or over-prediction is inverted. In this article, we have decided to use the ASHRAE Guideline 14 criteria. $NMBE$ is also subject to cancellation errors; consequently, the use of this index alone is not recommended.

$$NMBE=\frac{1}{\overline{m}}·\frac{{\sum }_{i=1}^n(m_i-s_i)}{n-p}\times 100(\%)$$

(2)

$CV\left(RMSE\right)$ (Coefficient of Variation of the Root Mean Square Error) measures the variability of the errors between measured and simulated values. It “gives an indication of the model’s ability to predict the overall load shape that is reflected in the data” [19]. In this case, the value of p is suggested to be one [17,18]. It is not subject to cancellation errors, and hence, AHSRAE Guidelines, FEMP and IPMVP use it with $NMBE$ to verify the accuracy of the models (see

Table 1. Calibration criteria of the Federal Energy Management Program (FEMP), ASHRAE Guideline 14 and IPMVP.

Data Type	Index	FEMP Criteria [11,12]	ASHRAE Guideline 14 [8,9]	IPMVP [16]
Calibration criteria
Monthly criteria %	$NMBE$	±5	±5	±20
	$CV\left(RMSE\right)$	15	15	-
Hourly criteria %	$NMBE$	±10	±10	±5
	$CV\left(RMSE\right)$	30	30	20
Model recommendation
	$R^2$	-	>0.75	>0.75

).

$$CV\left(RMSE\right)=\frac{1}{\overline{m}}\sqrt{\frac{{\sum }_{i=1}^n{(m_i-s_i)}^2}{n-p}}\times 100(\%)$$

(3)

$R^2$ (coefficient of determination) indicates how close simulated values are to the regression line of the measured values. It is another statistical index commonly used to measure the uncertainty of the models. It is limited to between 0.00 and 1.00 where the upper value means that the simulated values match the measured ones perfectly and the lower ones do not. It is not a prescriptive value for calibrated models, but both the ASHRAE Handbook [20] and IPVMP [16] recommend that the value never be less than 0.75 for calibrated models.

$$R^2={\left(\frac{n·{\sum }_{i=1}^n{m}_i·s_i-{\sum }_{i=1}^n{m}_i·{\sum }_{i=1}^n{s}_i}{\sqrt{(n·{\sum }_{i=1}^n{m}_i^2-{\left({\sum }_{i=1}^n{m}_i\right)}^2)·(n·{\sum }_{i=1}^n{s}_i^2-{\left({\sum }_{i=1}^n{s}_i\right)}^2)}}\right)}^2$$

(4)

Other statistical indices not used by ASHRAE Guidelines, FEMP and IPVMP, but which have been found to be useful for calibration purposes [21,22,23,24,25,26,27] are the Goodness-Of-Fit index ($GOF$) and the cost function $f_i$.

$GOF$ (Goodness-Of-Fit) measures how well the simulated values fit the measured ones. Lower values mean lower dispersion, so the regression line of the model is closer to the real one. ASHRAE Research Project 1051-RP [23] explains this index in detail. Equation (5) shows a simplified version of the $GOF$ index where there is no guarantee of the weighting factors of $NMBE$ and $CV\left(RMSE\right)$. As can be seen, this statistical index combines the overall behavior of $CV\left(RMSE\right)$ and $NMBE$. Therefore, in order to obtain a good value of $GOF$, the other indices also need to be good.

$$GOF=\frac{\sqrt{2}}{2}·\sqrt{CV{\left(RMSE\right)}^2+NMB{E}^2}$$

(5)

$f_i$ is a cost function that takes into account the values of $CV\left(RMSE\right)$ and $R^2$ with equal weighting. $CV{\left(RMSE\right)}_i$ and $R_i^2$ are calculated for each time step, while $CV{\left(RMSE\right)}_{ini}$ and $R_{ini}^2$ are the initial values. As with $GOF$, low values mean that the simulated and measured values fit better.

$$f_i=0.5\times CV{\left(RMSE\right)}_i+0.5\times (1-R_i^2)\times \frac{CV{\left(RMSE\right)}_{ini}}{(1-R_{ini}^2)}$$

(6)

Table 1 summarizes the criteria of the three main documents to validate a model as calibrated.

3. Analysis and Magnitude of the Common Error

The common error made is the use of the abbreviation $MBE$ when referring to $NMBE$ to measure the accuracy of the models. It is a common mistake as discussed later (Section 5), which is not serious if only the abbreviation is affected. If the calculation of the uncertainty index is performed using the $MBE$ formula instead of $NMBE$, the error could be important.

Table 2. Importance of the use of the $NMBE$ uncertainty index to perform comparisons between models for calibration purposes.

Month	Measured (J)	Simulated (J)	Difference (J)	Measured (kWh)	Simulated (kWh)	Difference (kWh)
January	9.6506 × 109	9.6606 × 109	−1.00 × 107	2680.72	2683.50	−2.78
February	6.704 × 109	6.714 × 109	−1.00 × 107	1862.23	1865.01	−2.78
March	4.7898 × 109	4.7998 × 109	−1.00 × 107	1330.51	1333.28	−2.78
April	2.5153 × 109	2.5253 × 109	−1.00 × 107	698.68	701.46	−2.78
May	5.4644 × 108	5.5644 × 108	−1.00 × 107	151.79	154.57	−2.78
June	2.5425 × 108	2.6425 × 108	−1.00 × 107	70.62	73.40	−2.78
July	1.5667 × 109	1.5767 × 109	−1.00 × 107	435.20	437.98	−2.78
August	1.1871 × 109	1.1971 × 109	−1.00 × 107	329.75	332.53	−2.78
September	2.7977 × 108	2.8977 × 108	−1.00 × 107	77.71	80.49	−2.78
October	1.1759 × 109	1.1859 × 109	−1.00 × 107	326.64	329.42	−2.78
November	6.0358 × 109	6.0458 × 109	−1.00 × 107	1676.61	1679.39	−2.78
December	9.3627 × 109	9.3727 × 109	−1.00 × 107	2600.76	2603.54	−2.78
		$MBE$	−1.00 × 107		$MBE$	−2.78
	Mean (Measured)		3.6724 × 109	Mean (Measured)		1020.10
		$NMBE$	−0.27%		$NMBE$	−0.27%

shows an invented example to analyze the magnitude of this error: the use of the $MBE$ formula for calibration purposes. It is a comparison between the same measurements, but using different units (J on the left and kWh on the right). In both cases, the measured and simulated values correspond to the energy demand (heating and cooling) of a building. In all the cases, the simulated values (as it is a fictitious example) are the result of adding 1.00 × 10⁷ J (2.78 kWh) to the measured values. As can be seen, there is a major difference between the values of $MBE$ because its values are not normalized. The measurement of this index is completely different when referring to a large value (J) or a small one (kWh). Once the values are weighted by their mean, so they are normalized ($NMBE$), they are perfectly comparable.

As shown in Section 5, in most cases, it is only a mistake with the abbreviation of the uncertainty index used, but the number of cases involved is a clear sign of the serious confusion about how to know if a model is calibrated. The main problem is that these errors might produce false assurance that the model used is calibrated if the uncertainty index is improperly calculated.

4. Origin and Spread of the Error

The origin of these three main protocols and guidelines dates from 1996, when the North American Measurement and Verification Protocol (NEMVP) [28] was published. The aim of this protocol is to reduce the consumption of energy and water by quantifying both the performance of the Energy Conservation Measures (ECMs) and their energy savings. In 1997, due to widespread interest outside North America, the first version of the IPMVP protocol was published [29]. Later, in 2000, the first version of FEMP was published [10]. It is based on the IPVMP protocol, but focusing on federal energy projects. From then on, both FEMP and IPMVP have been updated in different versions, although the core of the documents is the same. There is a fourth protocol produced by the Australasian Energy Performance Contracting Association for the Innovation Access Program [30] that is almost a copy of IPMVP, and for this reason, it is not included in this analysis.

In 2002, ASHRAE published its Guideline 14 regarding “Measurement of Energy and Demand Savings”. Its intention is “to provide guidance on minimum acceptable levels of performance for determining energy and demand savings, using measurements”. It is more technical than the other documents, and as a result, the majority of the scientific community uses this document in their research. Because of that, we will analyze the mistakes of this protocol related to “Whole Building Calibrated Simulation Approach (Calibrated Simulation)”. The following is a brief summary of the use of $NMBE$ and $CV\left(RMSE\right)$ in ASHRAE Guideline 14-2002 in which an abbreviation error of $NMBE$ is analyzed in detail (in order of appearance in the document):

• Section 5.2.11.3 Modeling Uncertainty establishes the correct formulas of $NMBE$ and $CV\left(RMSE\right)$ and explains that the Guideline “uses the following (…) indices to represent how well a mathematical model describes the variability in measured data” for calibrated simulations.
• Table 5-2 Path Specific Compliance Requirements sets out the minimal requirements for three specific approaches: (a) whole building, (b) retrofit isolation and (c) whole-building calibrated simulation. Points 7 and 9 of this table (baseline model uncertainty and uncertainty analysis respectively) explain that the uncertainty analysis of the calibrated simulation is required, establishing its limits in “Note 2”. The limits established in this note are those summarized in Table 1. In the note, the abbreviation of $CV\left(RMSE\right)$ has been omitted. It is assumed to be the first values. Figure 1 is a fragment of this table.
• Section 5.3.2.4 Whole Building Calibrated Simulation Performance Path, Point “f” again establishes the limits of $NMBE$ and $CV\left(RMSE\right)$.
• Table 5-3 Path Specific Requirements of the Measurement and Verification Plan. In this table (point “3”), the mistake with the abbreviation of $NMBE$ starts: “$MBE$ and $CV\left(RMSE\right)$ of computer baseline model relative to calibration data”. From this point to the end of the document, the abbreviation of $NMBE$ is incorrectly named as $MBE$ when referring to whole-building calibrated simulation limits. Figure 2 is a fragment of this table.
• Section 6.3.3.4.1 Calibrate to Monthly Utility Bills and Spot Measurements. In the last paragraph, it explains the acceptable tolerances based on $MBE$ and $CV\left(RMSE\right)$.
• Section 6.3.3.4.2.2 Statistical Comparison Techniques. It explains the statistical indices used, but in the definition of $MBE$, the $NMBE$ is being described.

ASHRAE Guideline 14-2002 has two errata sheets published [31,32], but they focus on other mistakes. In 2014, ASHRAE published a new version of its Guideline 14 [8]. It is a shorter document that refers to IPMVP 2012 [16] for additional information. The abbreviation errors related to $MBE\to NMBE$ are the same as in the former version, and only the number of the sections has been changed (Section 5.2.11.3 → 4.2.11.3, Table 5-2 → 4-2, Section 5.3.2.4 → 4.3.2.4, Table 5-3 → 4-3, Section 6.3.3.4.1 → 5.3.3.3.10, Section 6.3.3.4.2.2 → Annex C2).

Relating to IPMVP documents, the explanation and calculation of uncertainty is located in its Appendix B of Volume I. However, the document contents vary depending on the year published. IPMVP 2002 [13] establishes the limits of the uncertainty indices $MBE$ and $CV\left(RMSE\right)$ in Section 3.4.4.2 Option D: Calibration (see Table 1) and describes a correct $NMBE$ formula named as $MBE$ in its introduction of Appendix B. However, later versions of the IPMVP document (2009 [14], 2010 [15], 2012 [16]) refer to AHSRAE Guideline 14-2002 to obtain technical details on calibration simulation and only describe a correct formula of $MBE$ in its Appendix B-2.2.2 Standard Error of the Estimate. The error is guaranteed if the $MBE$ formula of Versions 2009, 2010, 2012 is used with the limits of Version 2002.

Finally, the FEMP document (Versions 2.2 [10], 3.0 [11], 4.0 [12]) uses a different formula of $MBE$ (see Equation (7)). It is a formula often used in articles, as we shall see below (Section 5).

$$MBE(\%)=\frac{{\sum }_{i=1}^n(m_i-s_i)}{{\sum }_{i=1}^n\left(m_i\right)}\times 100(\%)$$

(7)

Indeed, it is a correct $NMBE$ formula, but it is simplified. Equation (8) explains it. The value of p = 0 is in accordance with suggestions from Reddy and Maor 2006 (ASHRAE 1051-RP [23]).

$$NMBE=\frac{1}{\overline{m}}·\frac{{\sum }_{i=1}^n(m_i-s_i)}{n-p}\times 100(\%)⟶\overline{m}=\frac{{\sum }_{i=1}^n\left(m_i\right)}{n}$$

(8)

$$NMBE=\frac{n}{{\sum }_{i=1}^n\left(m_i\right)}·\frac{{\sum }_{i=1}^n(m_i-s_i)}{n}\times 100(\%)\to \frac{{\sum }_{i=1}^n(m_i-s_i)}{{\sum }_{i=1}^n\left(m_i\right)}\times 100(\%)\to MBE(\%)$$

Another difference between the versions of the FEMP document is that Version 2.2 subtracts measured values ($m_i$) from simulated ones ($s_i$) and in the successive versions does the opposite (see Section 2). Finally, the acceptable calibration tolerances are more restrictive in Versions 3.0 and 4.0 than in Version 2.2. These limits are summarized in Table 1.

5. References and Journals Affected

In this section, we are going to look at some examples where the use of the abbreviation $MBE$ is erroneous. Such examples have been taken from the most important journals and documents in the energy field, in particular the documents that use building calibrated models in their research. Obviously, in this search, there is a wide variety of articles, reports and theses where $NMBE$ is used correctly [17,18,19,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58].

Table 3. Articles and other documents with errors identified.

Title	Year	Journal/Report/Thesis	Ref.	Statistical Indices Used	Error	Source	About
Evaluation of overall thermal transfer value (OTTV) for commercial buildings constructed with green roof	2013	Applied energy	[59]	MBE, RMSE	1	-	Building energy simulations
A model calibration framework for simultaneous multi-level building energy simulation	2015	Applied Energy	[60]	MBE, CVRMSE	1	ASHRAE 14-2002, IPMVP, FEMP	Building energy calibration
Expanding Inter-Building Effect modeling to examine primary energy for lighting	2014	Energy and Buildings	[61]	MBE, RMSE	1	ASHRAE 14-2002	Building energy simulation
Estimating the impact of climate change and local operational procedures on the energy use in several supermarkets throughout Great Britain	2016	Energy and Buildings	[62]	MBE, CVRMSE, R${}^2$	1	[63]	Energy consumption
Assessment of SUNY version 3 global horizontal and direct normal solar irradiance in Canada	2012	Energy Procedia	[64]	MBE, RMSE	1	-	Solar radiation calculations
Generation of typical meteorological years using genetic algorithm for different energy systems	2016	Renewable Energy	[65]	MBE, RMSE	1	-	Weather calculations
Simple model for estimating global solar radiation	1985	Solar and Wind Technology	[66]	RMSE, MBE	1	-	Solar radiation calculations
A transdisciplinary approach on the energy efficient retrofitting of a historic building in the Aegean Region of Turkey	2015	Energy and Buildings	[67]	MBE, RMSE, CVRMSE	2	ASHRAE 14-2002	Building energy retrofitting
Ongoing commissioning of water-cooled electric chillers using benchmarking models	2012	Applied energy	[68]	MBE, CVRMSE, RMSE, R${}^2$	3	ASHRAE 14-2002	Commissioning of electric chillers
Operation and control strategies for multi-storey double skin facades during the heating season	2012	Energy and Buildings	[69]	MBE, RMSE, R${}^2$	3	ASHRAE 14-2002	Energy strategies
Modeling hourly and daily fractions of UV, PAR and NIR to global solar radiation under various sky conditions at Botucatu, Brazil	2009	Applied Energy	[70]	MBE, RMSE	4	-	Solar radiation calculations
The role of clouds in improving the regression model for hourly values of diffuse solar radiation	2012	Applied Energy	[71]	MBE, AIC, R${}^2$, RMSE	4	-	Solar radiation in relation with clouds
Human-based energy retrofits in residential buildings: A cost-effective alternative to traditional physical strategies	2014	Applied Energy	[72]	MBE, CVRMSE	4	ASHRAE 14-2002
Historical buildings: Multidisciplinary approach to structural/energy diagnosis and performance assessment	2015	Applied Energy	[73]	MBE, CVRMSE	4	ASHRAE 14-2002	Structural/energy diagnosis of a building
Development of a model predictive control framework through real-time building energy management system data	2015	Applied Energy	[74]	MBE, CVRMSE	4	ASHRAE 14-2002	Model predictive control
Why is the reliability of building simulation limited as a tool for evaluating energy conservation measures?	2015	Applied Energy	[75]	MBE, CVRMSE	4	ASHRAE 14-2002, IPMVP, FEMP	Limits of energy simulation
An EnergyPlus whole building energy model calibration method for office buildings using occupant behavior data mining and empirical data	2014	Carnegie Mellon University, ASHRAE/IBPSA-USA	[76]	MBE, CVRMSE	4	ASHRAE 14-2002	Building energy calibration
Development and validation of a Radiance model for a translucent panel	2006	Energy and Buildings	[77]	MBE, RMSE	4	-	Daylight study
Heating system performance estimation using optimization tool and BEMS data	2008	Energy and Buildings	[78]	MBE, CVRMSE	4	ASHRAE 14-2002	Heating systems
Calibrating whole building energy models: An evidence-based methodology	2011	Energy and Buildings	[79]	MBE, CVRMSE	4	ASHRAE 14-2002, IPMVP, FEMP	Building energy calibration
A comprehensive analysis of the impact of occupancy parameters in energy simulation of office buildings	2012	Energy and Buildings	[80]	MBE	4	FEMP 3.0, ASHRAE 14-2007 (error, is 2002)	Occupancy behavior and energy consumption
Analysis of building energy consumption parameters and energy savings measurement and verification by applying eQUEST software	2013	Energy and Buildings	[81]	MBE, RMSE	4	-	Building Energy Consumption
On-site monitoring and dynamic simulation of a low energy house heated by a pellet boiler	2016	Energy and Buildings	[82]	RMSD, MBE, CVRMSD, f cost function	4	ASHRAE 14-2002	Building energy calibration
Modelling of a Multi-purpose Commercial Building for Demand Response Analysis	2015	Energy Procedia	[83]	MBE	4	-	Demand Response
Calibration of Building Energy Simulation Models Based on Optimization: A Case Study	2015	Energy Procedia	[84]	MBE, CVRMSE	4	ASHRAE 14-2002	Building energy calibration
Experimental and numerical study on thermal performance of new cool clay tiles in residential buildings in Europe	2015	Energy Procedia	[85]	MBE, CVRMSE	4	ASHRAE 14-2002	Building energy simulation
A review of the CIE general sky classification approaches	2014	Renewable and Sustainable Energy Reviews	[86]	MBE, CVRMSE	4	-	Sky classification
A Procedure to Perform Multi-Objective Optimization for Sustainable Design of Buildings	2016	Energies	[87]	MBE, CVRMSE	5	-	Optimization of buildings
The all-sky meteorological radiation model: proposed improvements	2006	Applied energy	[88]	MBE, RMSE, R${}^2$	6	-	Solar radiation calculations
Model calibration for building energy efficiency simulation	2014	Applied energy	[89]	MBE, CVRMSE	7	ASHRAE 14-2002	Building energy calibration
Evaluation of “Autotune” calibration against manual calibration of building energy models	2016	Applied energy	[90]	MBE, CVRMSE	7	ASHRAE 14-2002, IPMVP, FEMP	Building energy calibration
Office building cooling load reduction using thermal analysis method–a case study	2016	Applied Energy	[91]	MBE, CVRMSE	7	ASHRAE 14-2002	Building energy simulation
Methodology of the cost-optimality for improving the indoor thermal environment during the warm season. Presentation of the method and application to a new multi-storey building in Berlin	2017	Applied Energy	[92]	MBE	7	FEMP 3.0	Improve indoor conditions
ESL-TR-94/07-01, Instructions for “Great Energy Predictor Shootout II: Measuring Retrofit Energy Savings”	1994	ASHRAE	[93]	MBE, CVRMSE	7	-	-
Guideline 14-2002, Measurement of Energy and Demand Savings	2012	ASHRAE	[9]	NMBE, MBE, CVRMSE	7	-	ASHRAE Guideline
ANN-Based Prediction and Optimization of Cooling System in Hotel Rooms	2015	Energies	[94]	MBE	7	-	Building energy simulation
Study on Auto-DR and pre-cooling of commercial buildings with thermal mass in California	2010	Energy and Buildings	[95]	MBE, CVRMSE, RMSE	7	ASHRAE 14-2002	Building energy simulation
Calibrating whole building energy models: Detailed case study using hourly measured data	2011	Energy and Buildings	[96]	MBE, CVRMSE	7	ASHRAE 14-2002	Building energy calibration
Optimization of an envelope retrofit strategy for an existing office building	2012	Energy and Buildings	[97]	MBE, RMSE	7	ASHRAE 14-2002, IPMVP, FEMP	Building energy optimization
Building hourly thermal load prediction using an indexed ARX model	2012	Energy and Buildings	[98]	MBE, CV, EEP, $\sigma$	7	-	Building thermal load prediction
Analysis of the impact of using synthetic data correlated with measured data on the calibrated as-built simulation of a commercial building	2013	Energy and Buildings	[99]	MBE, CVRMSE	7	ASHRAE 14-2002	Building energy calibration
Calibrated simulation of a public library HVAC system with a ground-source heat pump and a radiant floor using TRNSYS and GenOpt	2015	Energy and Buildings	[100]	MBE, CVRMSE	7	ASHRAE 14-2002	Building energy calibration
Energy saving potential through Energy Conservation Building Code and advance energy efficiency measures in hotel buildings of Jaipur City, India	2015	Energy and Buildings	[90]	MBE, CVRMSE	7	IPMVP	Energy efficiency
A simplified PEM fuel cell model for building cogeneration applications	2015	Energy and Buildings	[101]	MBE, CVRMSE	7	-	PEM fuel cell model
Application of a simplified thermal network model for real-time thermal load estimation	2015	Energy and Buildings	[102]	MBE, CVRMSE	7	FEMP 3.0	Building energy simulation
Building model calibration using energy and environmental data	2015	Energy and Buildings	[103]	MBE, CVRMSE	7	ASHRAE 14-2002	Building energy calibration
Analysis of energy efficiency retrofit scheme for hotel buildings using eQuest software: A case study from Tianjin, China	2015	Energy and Buildings	[104]	MBE, CVRMSE	7	ASHRAE 14-2002, IPMVP, FEMP	Building energy retrofitting
A process for developing deep energy retrofit strategies for single-family housing typologies: Three Toronto case studies	2016	Energy and Buildings	[105]	NMBE, CVRMSE	7	ASHRAE 14-2002	Energy Retrofits
Development of a new multi-stage building energy model calibration methodology and validation in a public library	2017	Energy and Buildings	[106]	MBE, CVRMSE	7	ASHRAE 14-2002	Building calibration
A combination of Heliosat-1 and Heliosat-2 methods for deriving solar radiation from satellite images	2014	Energy Procedia	[107]	MAE, MAE(%), MBE, MBE(%), RMSE, RMSE(%)	7	-	Solar radiation calculations
Development of models for on-line diagnostic and energy assessment analysis of PV power plants: the study case of 1 MW Sicilian PV plant	2015	Energy Procedia	[108]	MBE, RMSE, nRMSE	7	-	PV energy analysis
Modelling and calibration of a domestic building using high-resolution monitoring data	2016	IBPSA	[109]	MBE, CVRMSE	7	ASHRAE 14-2002	Building calibration
Simulation assisted audit & Evidence based calibration methodology	2010	IEA-ECBCS Annex 53	[110]	MBE, RMSE, CVRMSE	7	ASHRAE 14-2002, IPMVP, FEMP	Energy calibration methodology
Computing global and diffuse solar hourly irradiation on clear sky. Review and testing of 54 models	2012	Renewable and Sustainable Energy Reviews	[111]	MBE, RMSE	7	-	Solar radiation calculations
A review of methods to match building energy simulation models to measured data	2014	Renewable and sustainable energy reviews	[63]	MBE	7	ASHRAE 14-2002, IPMVP, FEMP	Building Calibration
Modeling and analysis of the spatiotemporal variations of photosynthetically active radiation in China during 1961–2012	2015	Renewable and Sustainable Energy Reviews	[112]	RE, MBE, MABE, RMSE, R${}^2$	7	-	Photosynthetically active radiation
Investigation of the variability of photosynthetically active radiation in the Tibetan Plateau, China	2016	Renewable and Sustainable Energy Reviews	[113]	MBE, MABE, RMSE	7	-	Solar radiation
Calibrated whole building energy simulation: An evidence-based methodology	2011	Thesis	[114]	MBE, CVRMSE	7	ASHRAE 14-2002, IPMVP, FEMP	Building calibration

is an overview of these documents ordered by error type, journal/report/thesis and year published. It is interesting to analyze the source used in each document to perform the uncertainty analysis. As can be seen, ASHRAE Guideline 14 is the most frequently used source of information. It is also logical that the sources of all the reported references with subjects regarding solar radiation calculations, weather calculations, etc. (see the About field on Table 3), are not based on the main documents described in Section 4.

Below is an explanation of the different types of errors related to the $MBE$ and $NMBE$ uncertainty indices:

Error 1

$MBE$ index with formula, but expressed as percentage (%): In these cases, the $MBE$ is defined correctly using the formula, but in the text, it is used in terms of (%) without specifying how this conversion has been calculated. If the values of $MBE$ had been taken directly, it would be an error.

Error 2

$MBE$ index with formula, but expressed directly as (%) to verify uncertainty limits: This is an error if the value is not normalized.

Error 3

$MBE$ index with formula used directly as analysis criteria: The use of this value directly makes no sense due to the cancellation errors.

Error 4

$MBE$ index without formula and expressed as (%). It is not possible to verify if the data used are correct.

Error 5

$MBE$ index described with an incorrect formula.

Error 6

$MBE$ index is explained, but the explanation is incorrect.

Error 7

$NMBE$ index with formula, but named as $MBE$ or $MBE(\%)$: This is the most common error. The formula and the data are correct, but the abbreviation used is incorrect.

6. Conclusions

The wide range of documents affected in some way by this mistake highlights the existing misunderstanding about how to perform an uncertainty analysis for model calibration purposes. The lack of unity of the main documents together with the fact that each researcher uses different criteria for his/her uncertainty calculations depending on the sources consulted are the causes of this growing misunderstanding. Due to the great interest in the optimization field of energy consumption and hence on Measurement and Verification (M&V) protocols, the importance of using calibrated energy models is increasing. That is why it is important to clarify the requirements to measure the accuracy of building energy models.

With this research we wish to stress the necessity of unifying the uncertainty analysis criteria of the three main documents (ASHRAE Guideline 14 [8], IPMVP [16], FEMP [12]) to reduce the risk of possible confusion since they could produce erroneous calibrated energy models.