Next Article in Journal
Analysis and Design of a Wireless Power Transfer System with Dual Active Bridges
Previous Article in Journal
Energy Consumption Forecasting for University Sector Buildings
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Validation of Calibrated Energy Models: Common Errors

School of Architecture, University of Navarra, 31009 Pamplona, Spain
*
Author to whom correspondence should be addressed.
Energies 2017, 10(10), 1587; https://doi.org/10.3390/en10101587
Submission received: 20 September 2017 / Revised: 5 October 2017 / Accepted: 9 October 2017 / Published: 12 October 2017

Abstract

:
Nowadays, there is growing interest in all the smart technologies that provide us with information and knowledge about the human environment. In the energy field, thanks to the amount of data received from smart meters and devices and the progress made in both energy software and computers, the quality of energy models is gradually improving and, hence, also the suitability of Energy Conservation Measures (ECMs). For this reason, the measurement of the accuracy of building energy models is an important task, because once the model is validated through a calibration procedure, it can be used, for example, to apply and study different strategies to reduce its energy consumption in maintaining human comfort. There are several agencies that have developed guidelines and methodologies to establish a measure of the accuracy of these models, and the most widely recognized are: ASHRAE Guideline 14-2014, the International Performance Measurement and Verification Protocol (IPMVP) and the Federal Energy Management Program (FEMP). This article intends to shed light on these validation measurements (uncertainty indices) by focusing on the typical mistakes made, as these errors could produce a false belief that the models used are calibrated.

1. Introduction

One of the main focuses of governments’ policies in almost every country is the energy supply. Energy security has a direct impact on social welfare, the level of the economy and the safety of a country, and for this reason, all possible energy savings in all sectors are important. The International Energy Agency highlights that the buildings sector is responsible for over one-third of final energy consumption, and therefore, it is an important source of CO 2 emissions [1]. Fortunately, the energy savings potential of the buildings sector is also high (approximately 10% and 40% in hot climates), and for this reason, the optimization of its energy consumption is a key factor in the governments’ road map.
In this scenario, the Measurement and Verification (M&V) protocols are critical due to their capacity to evaluate the scope of each Energy Conservation Measure (ECM). Therefore, trustworthy energy saving calculations are crucial to encourage Energy Services Companies (ESCOs), building owners, utilities, etc., to make investments in energy projects to improve building energy efficiency and reduce CO 2 emissions.
The relentless advance of technology (the rising availability of “smart meters” cheaper, more accurate and better simulation software, powerful computers, etc.) is increasing this “trust” in energy saving calculations where energy models play a key role [2,3,4]. Nowadays, the whole-building approach of Building Energy Models (BEMs) is more common than the single measure approaches due to the increase levels of automation, lower costs, etc. [5,6,7]. The use of these energy models is not only to study ECMs, but also to use them as a Model Predictive Control (MPC), as software-as-a-service to track the energy consumption, or as a tool to study risks in building energy management for insurance companies, etc. The accuracy of the BEM used is a determining factor for all these applications; therefore, the use of calibrated models is required.
ASHRAE guidelines 14-2014 defines calibration as the: “(b) process of reducing the uncertainty of a model by comparing the predicted output of the model under a specific set of conditions to the actual measured data for the same set of conditions. (…) calibration includes following defined procedures that identify what parameters of the instrument, meter, or model may be adjusted, determining what is an acceptable level of accuracy or uncertainty, and documenting the process and results [8]”. As can be seen, a calibrated model is one that, under the same set of conditions, can reproduce the measured data; and its accuracy is measured by an uncertainty analysis.
This article analyzes the most common uncertainty indices used in the process of calibration (Section 2), but focuses mainly on both the typical mistakes made when using these indices and their magnitude (Section 3). Section 4 explains the origin of a common error in the use of these uncertainty indices and Section 5 its spread through different documents (journals, thesis, reports, etc.). Section 6 shows the conclusions. Finally, we wish to stress that we hope not to offend anyone cited in this paper because the mistakes highlighted are typical due to the unclear existing references.

2. Measuring Uncertainty

As ASHRAE Guideline 14-2002 and 14-2014 explain, the uncertainty analysis is the “(b) process of determining the degree of confidence in the true value when using measurement procedures and/or calculations” [8,9]. The three main sources that explain how to determine this “degree” of confidence, its uncertainty, are ASHRAE Guideline 14, FEMP [10,11,12] and IPVMP [13,14,15,16]. They use simplified methods to quantify the uncertainty in saving computations and, as ASHRAE Guideline 14 explains, others can be used if they are compliant with the Guideline (Subsection 5.2.11 [9] or 4.2.11 [8]).
The principal uncertainty indices used are: Normalized Mean Bias Error ( N M B E ), Coefficient of Variation of the Root Mean Square Error ( C V ( R M S E ) ) and coefficient of determination ( R 2 ). However, we start the explanation of these indices by defining the Mean Bias Error ( M B E ), due to its importance when the analysis and magnitude of the error is explained (Section 3).
M B E (Mean Bias Error), as its name indicates, is the average of the errors of a sample space. Generally, it is a good indicator of the overall behavior of the simulated data with regards to the regression line of the sample. In Equation (1), m i is the measured value, s i is the simulated one and n the number of measured data points. Positive values mean that the model under-predicts measured data, and a negative one means over-prediction. However, the main problem with this index is that it is subject to cancellation errors where the sum of positive and negative values could reduce the value of M B E . In fact, when a model is calibrated or near calibrated, the regression line of the sample is so close to the simulated one that the cancellation effect increases considerably.
M B E = i = 1 n ( m i s i ) n
N M B E (Normalized Mean Bias Error) is a normalization of the M B E index that is used to scale the results of M B E , making them comparable. It quantifies the M B E index by dividing it by the mean of measured values ( m ¯ ), giving the global difference between the real values and the predicted ones. In Equation (2), p is the number of adjustable model parameters, which, for calibration purposes, is suggested to be zero [17,18]. As in the case of M B E , positive and negative values mean the under- or over-prediction of this normalization. ASHRAE Guidelines [8,9] subtract measured values ( m i ) from simulated ones ( s i ) instead of FEMP [11,12] and IPMVP [16], which do the opposite. For this reason, the explanation of the under- or over-prediction is inverted. In this article, we have decided to use the ASHRAE Guideline 14 criteria. N M B E is also subject to cancellation errors; consequently, the use of this index alone is not recommended.
N M B E = 1 m ¯ · i = 1 n ( m i s i ) n p × 100 ( % )
C V ( R M S E ) (Coefficient of Variation of the Root Mean Square Error) measures the variability of the errors between measured and simulated values. It “gives an indication of the model’s ability to predict the overall load shape that is reflected in the data” [19]. In this case, the value of p is suggested to be one [17,18]. It is not subject to cancellation errors, and hence, AHSRAE Guidelines, FEMP and IPMVP use it with N M B E to verify the accuracy of the models (see Table 1).
C V ( R M S E ) = 1 m ¯ i = 1 n ( m i s i ) 2 n p × 100 ( % )
R 2 (coefficient of determination) indicates how close simulated values are to the regression line of the measured values. It is another statistical index commonly used to measure the uncertainty of the models. It is limited to between 0.00 and 1.00 where the upper value means that the simulated values match the measured ones perfectly and the lower ones do not. It is not a prescriptive value for calibrated models, but both the ASHRAE Handbook [20] and IPVMP [16] recommend that the value never be less than 0.75 for calibrated models.
R 2 = n · i = 1 n m i · s i i = 1 n m i · i = 1 n s i ( n · i = 1 n m i 2 ( i = 1 n m i ) 2 ) · ( n · i = 1 n s i 2 ( i = 1 n s i ) 2 ) 2
Other statistical indices not used by ASHRAE Guidelines, FEMP and IPVMP, but which have been found to be useful for calibration purposes [21,22,23,24,25,26,27] are the Goodness-Of-Fit index ( G O F ) and the cost function f i .
G O F (Goodness-Of-Fit) measures how well the simulated values fit the measured ones. Lower values mean lower dispersion, so the regression line of the model is closer to the real one. ASHRAE Research Project 1051-RP [23] explains this index in detail. Equation (5) shows a simplified version of the G O F index where there is no guarantee of the weighting factors of N M B E and C V ( R M S E ) . As can be seen, this statistical index combines the overall behavior of C V ( R M S E ) and N M B E . Therefore, in order to obtain a good value of G O F , the other indices also need to be good.
G O F = 2 2 · C V ( R M S E ) 2 + N M B E 2
f i is a cost function that takes into account the values of C V ( R M S E ) and R 2 with equal weighting. C V ( R M S E ) i and R i 2 are calculated for each time step, while C V ( R M S E ) i n i and R i n i 2 are the initial values. As with G O F , low values mean that the simulated and measured values fit better.
f i = 0.5 × C V ( R M S E ) i + 0.5 × ( 1 R i 2 ) × C V ( R M S E ) i n i ( 1 R i n i 2 )
Table 1 summarizes the criteria of the three main documents to validate a model as calibrated.

3. Analysis and Magnitude of the Common Error

The common error made is the use of the abbreviation M B E when referring to N M B E to measure the accuracy of the models. It is a common mistake as discussed later (Section 5), which is not serious if only the abbreviation is affected. If the calculation of the uncertainty index is performed using the M B E formula instead of N M B E , the error could be important.
Table 2 shows an invented example to analyze the magnitude of this error: the use of the M B E formula for calibration purposes. It is a comparison between the same measurements, but using different units (J on the left and kWh on the right). In both cases, the measured and simulated values correspond to the energy demand (heating and cooling) of a building. In all the cases, the simulated values (as it is a fictitious example) are the result of adding 1.00 × 107 J (2.78 kWh) to the measured values. As can be seen, there is a major difference between the values of M B E because its values are not normalized. The measurement of this index is completely different when referring to a large value (J) or a small one (kWh). Once the values are weighted by their mean, so they are normalized ( N M B E ), they are perfectly comparable.
As shown in Section 5, in most cases, it is only a mistake with the abbreviation of the uncertainty index used, but the number of cases involved is a clear sign of the serious confusion about how to know if a model is calibrated. The main problem is that these errors might produce false assurance that the model used is calibrated if the uncertainty index is improperly calculated.

4. Origin and Spread of the Error

The origin of these three main protocols and guidelines dates from 1996, when the North American Measurement and Verification Protocol (NEMVP) [28] was published. The aim of this protocol is to reduce the consumption of energy and water by quantifying both the performance of the Energy Conservation Measures (ECMs) and their energy savings. In 1997, due to widespread interest outside North America, the first version of the IPMVP protocol was published [29]. Later, in 2000, the first version of FEMP was published [10]. It is based on the IPVMP protocol, but focusing on federal energy projects. From then on, both FEMP and IPMVP have been updated in different versions, although the core of the documents is the same. There is a fourth protocol produced by the Australasian Energy Performance Contracting Association for the Innovation Access Program [30] that is almost a copy of IPMVP, and for this reason, it is not included in this analysis.
In 2002, ASHRAE published its Guideline 14 regarding “Measurement of Energy and Demand Savings”. Its intention is “to provide guidance on minimum acceptable levels of performance for determining energy and demand savings, using measurements”. It is more technical than the other documents, and as a result, the majority of the scientific community uses this document in their research. Because of that, we will analyze the mistakes of this protocol related to “Whole Building Calibrated Simulation Approach (Calibrated Simulation)”. The following is a brief summary of the use of N M B E and C V ( R M S E ) in ASHRAE Guideline 14-2002 in which an abbreviation error of N M B E is analyzed in detail (in order of appearance in the document):
  • Section 5.2.11.3 Modeling Uncertainty establishes the correct formulas of N M B E and C V ( R M S E ) and explains that the Guideline “uses the following (…) indices to represent how well a mathematical model describes the variability in measured data” for calibrated simulations.
  • Table 5-2 Path Specific Compliance Requirements sets out the minimal requirements for three specific approaches: (a) whole building, (b) retrofit isolation and (c) whole-building calibrated simulation. Points 7 and 9 of this table (baseline model uncertainty and uncertainty analysis respectively) explain that the uncertainty analysis of the calibrated simulation is required, establishing its limits in “Note 2”. The limits established in this note are those summarized in Table 1. In the note, the abbreviation of C V ( R M S E ) has been omitted. It is assumed to be the first values. Figure 1 is a fragment of this table.
  • Section 5.3.2.4 Whole Building Calibrated Simulation Performance Path, Point “f” again establishes the limits of N M B E and C V ( R M S E ) .
  • Table 5-3 Path Specific Requirements of the Measurement and Verification Plan. In this table (point “3”), the mistake with the abbreviation of N M B E starts: “ M B E and C V ( R M S E ) of computer baseline model relative to calibration data”. From this point to the end of the document, the abbreviation of N M B E is incorrectly named as M B E when referring to whole-building calibrated simulation limits. Figure 2 is a fragment of this table.
  • Section 6.3.3.4.1 Calibrate to Monthly Utility Bills and Spot Measurements. In the last paragraph, it explains the acceptable tolerances based on M B E and C V ( R M S E ) .
  • Section 6.3.3.4.2.2 Statistical Comparison Techniques. It explains the statistical indices used, but in the definition of M B E , the N M B E is being described.
ASHRAE Guideline 14-2002 has two errata sheets published [31,32], but they focus on other mistakes. In 2014, ASHRAE published a new version of its Guideline 14 [8]. It is a shorter document that refers to IPMVP 2012 [16] for additional information. The abbreviation errors related to M B E N M B E are the same as in the former version, and only the number of the sections has been changed (Section 5.2.11.3 → 4.2.11.3, Table 5-2 → 4-2, Section 5.3.2.4 → 4.3.2.4, Table 5-3 → 4-3, Section 6.3.3.4.1 → 5.3.3.3.10, Section 6.3.3.4.2.2 → Annex C2).
Relating to IPMVP documents, the explanation and calculation of uncertainty is located in its Appendix B of Volume I. However, the document contents vary depending on the year published. IPMVP 2002 [13] establishes the limits of the uncertainty indices M B E and C V ( R M S E ) in Section 3.4.4.2 Option D: Calibration (see Table 1) and describes a correct N M B E formula named as M B E in its introduction of Appendix B. However, later versions of the IPMVP document (2009 [14], 2010 [15], 2012 [16]) refer to AHSRAE Guideline 14-2002 to obtain technical details on calibration simulation and only describe a correct formula of M B E in its Appendix B-2.2.2 Standard Error of the Estimate. The error is guaranteed if the M B E formula of Versions 2009, 2010, 2012 is used with the limits of Version 2002.
Finally, the FEMP document (Versions 2.2 [10], 3.0 [11], 4.0 [12]) uses a different formula of M B E (see Equation (7)). It is a formula often used in articles, as we shall see below (Section 5).
M B E ( % ) = i = 1 n ( m i s i ) i = 1 n ( m i ) × 100 ( % )
Indeed, it is a correct N M B E formula, but it is simplified. Equation (8) explains it. The value of p = 0 is in accordance with suggestions from Reddy and Maor 2006 (ASHRAE 1051-RP [23]).
N M B E = 1 m ¯ · i = 1 n ( m i s i ) n p × 100 ( % ) m ¯ = i = 1 n ( m i ) n
N M B E = n i = 1 n ( m i ) · i = 1 n ( m i s i ) n × 100 ( % ) i = 1 n ( m i s i ) i = 1 n ( m i ) × 100 ( % ) M B E ( % )
Another difference between the versions of the FEMP document is that Version 2.2 subtracts measured values ( m i ) from simulated ones ( s i ) and in the successive versions does the opposite (see Section 2). Finally, the acceptable calibration tolerances are more restrictive in Versions 3.0 and 4.0 than in Version 2.2. These limits are summarized in Table 1.

5. References and Journals Affected

In this section, we are going to look at some examples where the use of the abbreviation M B E is erroneous. Such examples have been taken from the most important journals and documents in the energy field, in particular the documents that use building calibrated models in their research. Obviously, in this search, there is a wide variety of articles, reports and theses where N M B E is used correctly [17,18,19,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58].
Table 3 is an overview of these documents ordered by error type, journal/report/thesis and year published. It is interesting to analyze the source used in each document to perform the uncertainty analysis. As can be seen, ASHRAE Guideline 14 is the most frequently used source of information. It is also logical that the sources of all the reported references with subjects regarding solar radiation calculations, weather calculations, etc. (see the About field on Table 3), are not based on the main documents described in Section 4.
Below is an explanation of the different types of errors related to the M B E and N M B E uncertainty indices:
Error 1
M B E index with formula, but expressed as percentage (%): In these cases, the M B E is defined correctly using the formula, but in the text, it is used in terms of (%) without specifying how this conversion has been calculated. If the values of M B E had been taken directly, it would be an error.
Error 2
M B E index with formula, but expressed directly as (%) to verify uncertainty limits: This is an error if the value is not normalized.
Error 3
M B E index with formula used directly as analysis criteria: The use of this value directly makes no sense due to the cancellation errors.
Error 4
M B E index without formula and expressed as (%). It is not possible to verify if the data used are correct.
Error 5
M B E index described with an incorrect formula.
Error 6
M B E index is explained, but the explanation is incorrect.
Error 7
N M B E index with formula, but named as M B E or M B E ( % ) : This is the most common error. The formula and the data are correct, but the abbreviation used is incorrect.

6. Conclusions

The wide range of documents affected in some way by this mistake highlights the existing misunderstanding about how to perform an uncertainty analysis for model calibration purposes. The lack of unity of the main documents together with the fact that each researcher uses different criteria for his/her uncertainty calculations depending on the sources consulted are the causes of this growing misunderstanding. Due to the great interest in the optimization field of energy consumption and hence on Measurement and Verification (M&V) protocols, the importance of using calibrated energy models is increasing. That is why it is important to clarify the requirements to measure the accuracy of building energy models.
With this research we wish to stress the necessity of unifying the uncertainty analysis criteria of the three main documents (ASHRAE Guideline 14 [8], IPMVP [16], FEMP [12]) to reduce the risk of possible confusion since they could produce erroneous calibrated energy models.

Acknowledgments

We would like to thank the reviewer of our article “Genetic algorithm for building envelope calibration” [21], because he/she did not understand why we used M B E as the uncertainty index to validate our energy model. In our case, we were using the correct formula of N M B E , but were calling it M B E (Error 7) (see Section 5). After explaining the mistake, the article was accepted.

Author Contributions

Both authors have analyzed the common error and verified the existing misunderstanding in the calibration area. Germán Ramos Ruiz has analyzed the references that supports this study and has written the manuscript. Both authors have revised and verified all of the manuscript before sending it to the journal.

Conflicts of Interest

The authors declare no conflicts of interest.

Abbreviations

The following abbreviations are used in this manuscript:
ASHRAEAmerican Society of Heating, Refrigerating and Air-Conditioning Engineers
BEMsBuilding Energy Models
C V ( R M S E ) Coefficient of Variation of the Root Mean Square Error
ECMsEnergy Conservation Measures
ESCOsEnergy Services Companies
FEMPFederal Energy Management Program
GOFGoodness-f-Fit
IPMVPInternational Performance Measurement and Verification Protocol
M B E Mean Bias Error
MPCModel Predictive Control
M&VMeasurement and Verification
NEMVPNorth American Measurement and Verification Protocol
N M B E Normalized Mean Bias Error

References

  1. Dulac, J.; LaFrance, M.; Trudeau, N.; Yamada, H. Transition to Sustainable Buildings: Strategies and Oppourtunities to 2050; OECD: Billancourt, France, 2013. [Google Scholar]
  2. Crawley, D.B.; Lawrie, L.K.; Winkelmann, F.C.; Buhl, W.F.; Huang, Y.J.; Pedersen, C.O.; Strand, R.K.; Liesen, R.J.; Fisher, D.E.; Witte, M.J.; et al. EnergyPlus: Creating a new-generation building energy simulation program. Energy Build. 2001, 33, 319–331. [Google Scholar] [CrossRef]
  3. Guglielmetti, R.; Macumber, D.; Long, N. OpenStudio: An open source integrated analysis platform. In Proceedings of the 12th Conference of International Building Performance Simulation Association, Sydney, Australia, 14–16 November 2011. [Google Scholar]
  4. Zhang, Y. Use jEPlus as an efficient building design optimisation tool. In Proceedings of the CIBSE ASHRAE Technical Symposium, London, UK, 18–19 April 2012; pp. 18–19. [Google Scholar]
  5. Judkoff, R.; Wortman, D.; O’doherty, B.; Burch, J. Methodology for Validating Building Energy Analysis Simulations; Technical Report; National Renewable Energy Laboratory (NREL): Golden, CO, USA, 2008.
  6. Salvalai, G.; Pfafferott, J.; Jacob, D. Validation of a low-energy whole building simulation model. IBPSA-USA J. 2010, 4, 32–39. [Google Scholar]
  7. Salvalai, G. Implementation and validation of simplified heat pump model in IDA-ICE energy simulation environment. Energy Build. 2012, 49, 132–141. [Google Scholar] [CrossRef]
  8. American Society of Heating, Ventilating, and Air Conditioning Engineers (ASHRAE). Guideline 14-2014, Measurement of Energy and Demand Savings; Technical Report; American Society of Heating, Ventilating, and Air Conditioning Engineers: Atlanta, GA, USA, 2014. [Google Scholar]
  9. American Society of Heating, Ventilating, and Air Conditioning Engineers (ASHRAE). Guideline 14-2002, Measurement of Energy and Demand Savings; Technical Report; American Society of Heating, Ventilating, and Air Conditioning Engineers: Atlanta, GA, USA, 2002. [Google Scholar]
  10. Schiller, S.R.; Jump, D.A.; Franconi, E.M.; Stetz, M.; Geanacopoulos, A. M&V Guidelines: Measurement and Verification for Federal Energy Projects; Version 2.2, Technical Report; U.S. Department of Energy Federal Energy Management Program: Washington, DC, USA, 2000.
  11. Webster, L.; Bradford, J. M&V Guidelines: Measurement and Verification for Federal Energy Projects; Version 3.0, Technical Report; U.S. Department of Energy Federal Energy Management Program: Washington, DC, USA, 2008.
  12. Webster, L.; Bradford, J.; Sartor, D.; Shonder, J.; Atkin, E.; Dunnivant, S.; Frank, D.; Franconi, E.; Jump, D.; Schiller, S.; et al. M&V Guidelines: Measurement and Verification for Performance-Based Contracts; Version 4.0, Technical Report; U.S. Department of Energy Federal Energy Management Program: Washington, DC, USA, 2015.
  13. IPMVP Committee. International Performance Measurement and Verification Protocol: Concepts and Options for Determining Energy and Water Savings, Volume I; Technical Report; Efficiency Valuation Organization: Washington, DC, USA, 2002. [Google Scholar]
  14. Efficiency Valuation Organization. International Performance Measurement and Verification Protocol: Concepts and Options for Determining Energy and Water Savings, Volume I; Technical Report; Efficiency Valuation Organization: Washington, DC, USA, 2009. [Google Scholar]
  15. Efficiency Valuation Organization. International Performance Measurement and Verification Protocol: Concepts and Options for Determining Energy and Water Savings, Volume I; Technical Report; Efficiency Valuation Organization: Washington, DC, USA, 2010. [Google Scholar]
  16. Efficiency Valuation Organization. International Performance Measurement and Verification Protocol: Concepts and Options for Determining Energy and Water Savings, Volume I; Technical Report; Efficiency Valuation Organization: Washington, DC, USA, 2012. [Google Scholar]
  17. Reddy, T.A.; Maor, I.; Jian, S.; Panjapornporn, C. Procedures for Reconciling Computer-Calculated Results with Measured Energy Data; Technical Report; American Society of Heating, Refrigerating and Air-Conditioning Engineers: Atlanta, GA, USA, 2006. [Google Scholar]
  18. Robertson, J.; Polly, B.; Collis, J. Evaluation of Automated Model Calibration Techniques for Residential Building Energy Simulation; Technical Report, NREL Technical Report 5500-60127; National Renewable Energy Laboratory (NREL): Golden, CO, USA, 2013.
  19. Granderson, J.; Touzani, S.; Custodio, C.; Sohn, M.; Fernandes, S.; Jump, D. Assessment of Automated Measurement and Verification (M&V) Methods; Lawrence Berkeley National Laboratory: Berkeley, CA, USA, 2015.
  20. American Society of Heating, Refrigerating and Air Conditioning Engineers (ASHRAE). Handbook Fundamentals; American Society of Heating, Refrigerating and Air Conditioning Engineers: Atlanta, GA, USA, 2013; Volume 111. [Google Scholar]
  21. Ruiz, G.R.; Bandera, C.F.; Temes, T.G.A.; Gutierrez, A.S.O. Genetic algorithm for building envelope calibration. Appl. Energy 2016, 168, 691–705. [Google Scholar] [CrossRef]
  22. Ruiz, G.R.; Bandera, C.F. Analysis of uncertainty indices used for building envelope calibration. Appl. Energy 2017, 185, 82–94. [Google Scholar] [CrossRef]
  23. Reddy, T.; Maor, I. Procedures for Reconciling Computer-Calculated Results with Measured Energy Data; ASHRAE Research Project 1051-RP; American Society of Heating, Refrigerating and Air-Conditioning Engineers: Atlanta, GA, USA, 2006. [Google Scholar]
  24. Tahmasebi, F.; Mahdavi, A. A two-staged simulation model calibration approach to virtual sensors for building performance data. In Proceedings of the 13th Conference of International Building Performance Simulation Association, Chambéry, France, 25–30 August 2013; pp. 25–28. [Google Scholar]
  25. Tahmasebi, F.; Mahdavi, A. Monitoring-based optimization-assisted calibration of the thermal performance model of an office building. In Proceedings of the 1st International Conference on Architecture and Urban Design, Tirana, Albania, 19–21 April 2012. [Google Scholar]
  26. Tahmasebi, F.; Mahdavi, A. Optimization-based simulation model calibration using sensitivity analysis. In Simulace Budov a Techniky Prostredi; Sikula, O., Hirs, J., Eds.; International Building Performance Association—Česká Republika (IBPSA-CZ): Brno, Czech Republic, 2012. [Google Scholar]
  27. Tahmasebi, F.; Zach, R.; Schuß, M.; Mahdavi, A. Simulation model calibration: An optimization-based approach. BauSIM 2012 2012, 1, 386–391. [Google Scholar]
  28. Federal and State Agencies and Experts in the Energy and Efficiency Industries in America, Canada, and Mexico. North American Energy Measurement and Verification Protocol. Version 1.0; Technical Report; U.S. Department of Energy: Germantown, MD, USA, 1996.
  29. Efficiency Valuation Organization History Organization. Available online: https://evo-world.org/en/about-en/history-mainmenu-en (accessed on 10 October 2017).
  30. Australian Energy Performance Contracting Association. A Best Practice Guide to Measurement and Verification of Energy Savings; Technical Report; Australian Energy Performance Contracting Association: West Melbourne, Australia, 2004.
  31. American Society of Heating, Ventilating, and Air Conditioning Engineers (ASHRAE). Errata Sheet for ASHRAE Guideline 14-2002, Measurement of Energy and Demand Savings; American Society of Heating, Ventilating, and Air Conditioning Engineers: Atlanta, GA, USA, 2008. [Google Scholar]
  32. American Society of Heating, Ventilating, and Air Conditioning Engineers (ASHRAE). Errata Sheet for ASHRAE Guideline 14-2002, Measurement of Energy and Demand Savings (Downloaded Version Only); American Society of Heating, Ventilating, and Air Conditioning Engineers: Atlanta, GA, USA, 2008. [Google Scholar]
  33. Almorox, J.; Hontoria, C.; Benito, M. Models for obtaining daily global solar radiation with measured air temperature data in Madrid (Spain). Appl. Energy 2011, 88, 1703–1709. [Google Scholar] [CrossRef]
  34. Liu, X.; Xu, Y.; Zhong, X.; Zhang, W.; Porter, J.R.; Liu, W. Assessing models for parameters of the Ångström–Prescott formula in China. Appl. Energy 2012, 96, 327–338. [Google Scholar] [CrossRef]
  35. Garrett, A.; New, J.R. Suitability of ASHRAE Guideline 14 Metrics for Calibration. ASHRAE Trans. 2016, 122. [Google Scholar]
  36. Bhandari, M.; Shrestha, S.; New, J. Evaluation of weather datasets for building energy simulation. Energy Build. 2012, 49, 109–118. [Google Scholar] [CrossRef]
  37. Joe, J.; Choi, W.; Kwon, H.; Huh, J.H. Load characteristics and operation strategies of building integrated with multi-story double skin facade. Energy Build. 2013, 60, 185–198. [Google Scholar] [CrossRef]
  38. Ramamurthy, P.; Sun, T.; Rule, K.; Bou-Zeid, E. The joint influence of albedo and insulation on roof performance: An observational study. Energy Build. 2015, 93, 249–258. [Google Scholar] [CrossRef]
  39. Yin, R.; Kiliccote, S.; Piette, M.A. Linking measurements and models in commercial buildings: A case study for model calibration and demand response strategy evaluation. Energy Build. 2016, 124, 222–235. [Google Scholar] [CrossRef]
  40. Li, Y.; Rezgui, Y. A novel concept to measure envelope thermal transmittance and air infiltration using a combined simulation and experimental approach. Energy Build. 2017, 140, 380–387. [Google Scholar] [CrossRef]
  41. Rahimi, I.; Bakhtiari, B.; Qaderi, K.; Aghababaie, M. Calibration of Angstrom equation for estimating solar radiation using meta-heuristic harmony search algorithm (case study: Mashhad-East of Iran). Energy Procedia 2012, 18, 644–651. [Google Scholar] [CrossRef]
  42. Dal Pai, A.; Escobedo, J.F.; Dal Pai, E.; dos Santos, C.M. Estimation of hourly, daily and monthly mean diffuse radiation based on MEO shadowring correction. Energy Procedia 2014, 57, 1150–1159. [Google Scholar] [CrossRef]
  43. Kim, K.H.; Baltazar, J.C.; Haberl, J.S. Evaluation of meteorological base models for estimating hourly global solar radiation in Texas. Energy Procedia 2014, 57, 1189–1198. [Google Scholar] [CrossRef]
  44. Aman, S.; Simmhan, Y.; Prasanna, V.K. Holistic measures for evaluating prediction models in smart grids. IEEE Trans. Knowl. Data Eng. 2015, 27, 475–488. [Google Scholar] [CrossRef]
  45. Fu, Y.; Li, Z.; Zhang, H.; Xu, P. Using support vector machine to predict next day electricity load of public buildings with sub-metering devices. Procedia Eng. 2015, 121, 1016–1022. [Google Scholar] [CrossRef]
  46. Besharat, F.; Dehghan, A.A.; Faghih, A.R. Empirical models for estimating global solar radiation: A review and case study. Renew. Sustain. Energy Rev. 2013, 21, 798–821. [Google Scholar] [CrossRef]
  47. Khalil, S.A.; Shaffie, A. A comparative study of total, direct and diffuse solar irradiance by using different models on horizontal and inclined surfaces for Cairo, Egypt. Renew. Sustain. Energy Rev. 2013, 27, 853–863. [Google Scholar] [CrossRef]
  48. Despotovic, M.; Nedic, V.; Despotovic, D.; Cvetanovic, S. Review and statistical analysis of different global solar radiation sunshine models. Renew. Sustain. Energy Rev. 2015, 52, 1869–1880. [Google Scholar] [CrossRef]
  49. El Mghouchi, Y.; Ajzoul, T.; El Bouardi, A. Prediction of daily solar radiation intensity by day of the year in twenty-four cities of Morocco. Renew. Sustain. Energy Rev. 2016, 53, 823–831. [Google Scholar] [CrossRef]
  50. El Mghouchi, Y.; Ajzoul, T.; Taoukil, D.; El Bouardi, A. The most suitable prediction model of the solar intensity, on horizontal plane, at various weather conditions in a specified location in Morocco. Renew. Sustain. Energy Rev. 2016, 54, 84–98. [Google Scholar] [CrossRef]
  51. Abdel-Salam, M.; El-Dib, A.; Eissa, M. Prediction of ground-level solar radiation in Egypt. Renew. Energy 1991, 1, 269–276. [Google Scholar] [CrossRef]
  52. Zeroual, A.; Ankrim, M.; Wilkinson, A. The diffuse-global correlation: Its application to estimating solar radiation on tilted surfaces in Marrakesh, Morocco. Renew. Energy 1996, 7, 1–13. [Google Scholar] [CrossRef]
  53. Jarimi, H.; Bakar, M.N.A.; Othman, M.; Din, M.H. Bi-fluid photovoltaic/thermal (PV/T) solar collector: Experimental validation of a 2-D theoretical model. Renew. Energy 2016, 85, 1052–1067. [Google Scholar] [CrossRef]
  54. Kaplanis, S.; Kumar, J.; Kaplani, E. On a universal model for the prediction of the daily global solar radiation. Renew. Energy 2016, 91, 178–188. [Google Scholar] [CrossRef] [Green Version]
  55. Robertson, J.J. Application of Simulated Annealing Inversion to Automated Model Calibration for Residential Building Energy Simulation. Ph.D. Thesis, Colorado School of Mines, Golden, CO, USA, 2007. [Google Scholar]
  56. Ogando, A.; Cid, N.; Fernández, M. Energy Modelling and Automated Calibrations of Ancient Building Simulations: A Case Study of a School in the Northwest of Spain. Energies 2017, 10, 807. [Google Scholar] [CrossRef]
  57. Palmer, D.; Cole, I.; Betts, T.; Gottschalg, R. Interpolating and Estimating Horizontal Diffuse Solar Irradiation to Provide UK-Wide Coverage: Selection of the Best Performing Models. Energies 2017, 10, 181. [Google Scholar] [CrossRef]
  58. Dyke, C.; Van Den Wymelenberg, K.; Djunaedy, E.; Steciak, J. Comparing whole building energy implications of sidelighting systems with alternate manual blind control algorithms. Buildings 2015, 5, 467–496. [Google Scholar] [CrossRef]
  59. Chan, A.; Chow, T. Evaluation of overall thermal transfer value (OTTV) for commercial buildings constructed with green roof. Appl. Energy 2013, 107, 10–24. [Google Scholar] [CrossRef]
  60. Yang, Z.; Becerik-Gerber, B. A model calibration framework for simultaneous multi-level building energy simulation. Appl. Energy 2015, 149, 415–431. [Google Scholar] [CrossRef]
  61. Pisello, A.L.; Castaldo, V.L.; Taylor, J.E.; Cotana, F. Expanding Inter-Building Effect modeling to examine primary energy for lighting. Energy Build. 2014, 76, 513–523. [Google Scholar] [CrossRef]
  62. Braun, M.; Beck, S.; Walton, P.; Mayfield, M. Estimating the impact of climate change and local operational procedures on the energy use in several supermarkets throughout Great Britain. Energy Build. 2016, 111, 109–119. [Google Scholar] [CrossRef]
  63. Coakley, D.; Raftery, P.; Keane, M. A review of methods to match building energy simulation models to measured data. Renew. Sustain. Energy Rev. 2014, 37, 123–141. [Google Scholar] [CrossRef]
  64. Djebbar, R.; Morris, R.; Thevenard, D.; Perez, R.; Schlemmer, J. Assessment of SUNY version 3 global horizontal and direct normal solar irradiance in Canada. Energy Procedia 2012, 30, 1274–1283. [Google Scholar] [CrossRef]
  65. Chan, A. Generation of typical meteorological years using genetic algorithm for different energy systems. Renew. Energy 2016, 90, 1–13. [Google Scholar] [CrossRef]
  66. Wahab, M.A. Simple model for estimating global solar radiation. Sol. Wind Technol. 1985, 2, 69–71. [Google Scholar] [CrossRef]
  67. Şahin, C.D.; Arsan, Z.D.; Tuncoku, S.S.; Broström, T.; Akkurt, G.G. A transdisciplinary approach on the energy efficient retrofitting of a historic building in the Aegean Region of Turkey. Energy Build. 2015, 96, 128–139. [Google Scholar] [CrossRef]
  68. Monfet, D.; Zmeureanu, R. Ongoing commissioning of water-cooled electric chillers using benchmarking models. Appl. Energy 2012, 92, 99–108. [Google Scholar] [CrossRef]
  69. Choi, W.; Joe, J.; Kwak, Y.; Huh, J.H. Operation and control strategies for multi-storey double skin facades during the heating season. Energy Build. 2012, 49, 454–465. [Google Scholar] [CrossRef]
  70. Escobedo, J.F.; Gomes, E.N.; Oliveira, A.P.; Soares, J. Modeling hourly and daily fractions of UV, PAR and NIR to global solar radiation under various sky conditions at Botucatu, Brazil. Appl. Energy 2009, 86, 299–309. [Google Scholar] [CrossRef]
  71. Furlan, C.; De Oliveira, A.P.; Soares, J.; Codato, G.; Escobedo, J.F. The role of clouds in improving the regression model for hourly values of diffuse solar radiation. Appl. Energy 2012, 92, 240–254. [Google Scholar] [CrossRef]
  72. Pisello, A.L.; Asdrubali, F. Human-based energy retrofits in residential buildings: A cost-effective alternative to traditional physical strategies. Appl. Energy 2014, 133, 224–235. [Google Scholar] [CrossRef]
  73. Ascione, F.; Ceroni, F.; De Masi, R.F.; de Rossi, F.; Pecce, M.R. Historical buildings: Multidisciplinary approach to structural/energy diagnosis and performance assessment. Appl. Energy 2017, 185, 1517–1528. [Google Scholar] [CrossRef]
  74. Kwak, Y.; Huh, J.H.; Jang, C. Development of a model predictive control framework through real-time building energy management system data. Appl. Energy 2015, 155, 1–13. [Google Scholar] [CrossRef]
  75. Li, N.; Yang, Z.; Becerik-Gerber, B.; Tang, C.; Chen, N. Why is the reliability of building simulation limited as a tool for evaluating energy conservation measures? Appl. Energy 2015, 159, 196–205. [Google Scholar] [CrossRef]
  76. Lam, K.P.; Zhao, J.; Ydstie, B.E.; Wirick, J.; Qi, M.; Park, J. An EnergyPlus whole building energy model calibration method for office buildings using occupant behavior data mining and empirical data. In Proceedings of the ASHRAE/IBPSA-USA Building Simulation Conference, Atlanta, GA, USA; 2014; pp. 160–167. [Google Scholar]
  77. Reinhart, C.F.; Andersen, M. Development and validation of a Radiance model for a translucent panel. Energy Build. 2006, 38, 890–904. [Google Scholar] [CrossRef]
  78. Djuric, N.; Novakovic, V.; Frydenlund, F. Heating system performance estimation using optimization tool and BEMS data. Energy Build. 2008, 40, 1367–1376. [Google Scholar] [CrossRef]
  79. Raftery, P.; Keane, M.; O’Donnell, J. Calibrating whole building energy models: An evidence-based methodology. Energy Build. 2011, 43, 2356–2364. [Google Scholar] [CrossRef]
  80. Azar, E.; Menassa, C.C. A comprehensive analysis of the impact of occupancy parameters in energy simulation of office buildings. Energy Build. 2012, 55, 841–853. [Google Scholar] [CrossRef]
  81. Ke, M.T.; Yeh, C.H.; Jian, J.T. Analysis of building energy consumption parameters and energy savings measurement and verification by applying eQUEST software. Energy Build. 2013, 61, 100–107. [Google Scholar] [CrossRef]
  82. Carlon, E.; Schwarz, M.; Prada, A.; Golicza, L.; Verma, V.K.; Baratieri, M.; Gasparella, A.; Haslinger, W.; Schmidl, C. On-site monitoring and dynamic simulation of a low energy house heated by a pellet boiler. Energy Build. 2016, 116, 296–306. [Google Scholar] [CrossRef]
  83. Christantoni, D.; Flynn, D.; Finn, D.P. Modelling of a multi-purpose commercial building for demand response analysis. Energy Procedia 2015, 78, 2166–2171. [Google Scholar] [CrossRef]
  84. Monetti, V.; Davin, E.; Fabrizio, E.; André, P.; Filippi, M. Calibration of building energy simulation models based on optimization: A case study. Energy Procedia 2015, 78, 2971–2976. [Google Scholar] [CrossRef]
  85. Pisello, A.L.; Cotana, F. Experimental and numerical study on thermal performance of new cool clay tiles in residential buildings in Europe. Energy Procedia 2015, 75, 1393–1398. [Google Scholar] [CrossRef]
  86. Li, D.H.; Chau, T.; Wan, K.K. A review of the CIE general sky classification approaches. Renew. Sustain. Energy Rev. 2014, 31, 563–574. [Google Scholar] [CrossRef]
  87. Brunelli, C.; Castellani, F.; Garinei, A.; Biondi, L.; Marconi, M. A Procedure to Perform Multi-Objective Optimization for Sustainable Design of Buildings. Energies 2016, 9, 915. [Google Scholar] [CrossRef]
  88. Muneer, T.; Younes, S. The all-sky meteorological radiation model: Proposed improvements. Appl. Energy 2006, 83, 436–450. [Google Scholar] [CrossRef]
  89. Mustafaraj, G.; Marini, D.; Costa, A.; Keane, M. Model calibration for building energy efficiency simulation. Appl. Energy 2014, 130, 72–85. [Google Scholar] [CrossRef]
  90. Chaudhary, G.; New, J.; Sanyal, J.; Im, P.; O’Neill, Z.; Garg, V. Evaluation of “Autotune” calibration against manual calibration of building energy models. Appl. Energy 2016, 182, 115–134. [Google Scholar] [CrossRef]
  91. Nazi, W.I.W.M.; Royapoor, M.; Wang, Y.; Roskilly, A.P. Office building cooling load reduction using thermal analysis method—A case study. Appl. Energy 2017, 185, 1574–1584. [Google Scholar] [CrossRef]
  92. Ascione, F.; Böttcher, O.; Kaltenbrunner, R.; Vanoli, G.P. Methodology of the cost-optimality for improving the indoor thermal environment during the warm season. Presentation of the method and application to a new multi-storey building in Berlin. Appl. Energy 2017, 185, 1529–1541. [Google Scholar] [CrossRef]
  93. Haberl, J. S.; Kreider, J. F. Instructions for “The Great Energy Predictor Shootout II: Measuring Retrofit Energy Savings”. Energy Systems Laboratory, Texas A&M University. Department of Mechanical Engineering, Texas A&M University; Available online: http://hdl.handle.net/1969.1/2137 (accessed on 9 October 2017).
  94. Moon, J.W.; Kim, K.; Min, H. ANN-based prediction and optimization of cooling system in hotel rooms. Energies 2015, 8, 10775–10795. [Google Scholar] [CrossRef]
  95. Yin, R.; Xu, P.; Piette, M.A.; Kiliccote, S. Study on Auto-DR and pre-cooling of commercial buildings with thermal mass in California. Energy Build. 2010, 42, 967–975. [Google Scholar] [CrossRef]
  96. Raftery, P.; Keane, M.; Costa, A. Calibrating whole building energy models: Detailed case study using hourly measured data. Energy Build. 2011, 43, 3666–3679. [Google Scholar] [CrossRef]
  97. Güçyeter, B.; Günaydın, H.M. Optimization of an envelope retrofit strategy for an existing office building. Energy Build. 2012, 55, 647–659. [Google Scholar] [CrossRef]
  98. Yun, K.; Luck, R.; Mago, P.J.; Cho, H. Building hourly thermal load prediction using an indexed ARX model. Energy Build. 2012, 54, 225–233. [Google Scholar] [CrossRef]
  99. Song, S.; Haberl, J.S. Analysis of the impact of using synthetic data correlated with measured data on the calibrated as-built simulation of a commercial building. Energy Build. 2013, 67, 97–107. [Google Scholar] [CrossRef]
  100. Cacabelos, A.; Eguía, P.; Míguez, J.L.; Granada, E.; Arce, M.E. Calibrated simulation of a public library HVAC system with a ground-source heat pump and a radiant floor using TRNSYS and GenOpt. Energy Build. 2015, 108, 114–126. [Google Scholar] [CrossRef]
  101. Ham, S.W.; Jo, S.Y.; Dong, H.W.; Jeong, J.W. A simplified PEM fuel cell model for building cogeneration applications. Energy Build. 2015, 107, 213–225. [Google Scholar] [CrossRef]
  102. Ogunsola, O.T.; Song, L. Application of a simplified thermal network model for real-time thermal load estimation. Energy Build. 2015, 96, 309–318. [Google Scholar] [CrossRef]
  103. Royapoor, M.; Roskilly, T. Building model calibration using energy and environmental data. Energy Build. 2015, 94, 109–120. [Google Scholar] [CrossRef]
  104. Xing, J.; Ren, P.; Ling, J. Analysis of energy efficiency retrofit scheme for hotel buildings using eQuest software: A case study from Tianjin, China. Energy Build. 2015, 87, 14–24. [Google Scholar] [CrossRef]
  105. Jermyn, D.; Richman, R. A process for developing deep energy retrofit strategies for single-family housing typologies: Three Toronto case studies. Energy Build. 2016, 116, 522–534. [Google Scholar] [CrossRef]
  106. Cacabelos, A.; Eguía, P.; Febrero, L.; Granada, E. Development of a new multi-stage building energy model calibration methodology and validation in a public library. Energy Build. 2017, 146, 182–199. [Google Scholar] [CrossRef]
  107. Pagola, I.; Gastón, M.; Bernardos, A.; Fernández-Peruchena, C. A combination of Heliosat-1 and Heliosat-2 methods for deriving solar radiation from satellite images. Energy Procedia 2014, 57, 1037–1043. [Google Scholar] [CrossRef]
  108. Ventura, C.; Tina, G.M. Development of models for on-line diagnostic and energy assessment analysis of PV power plants: The study case of 1 MW Sicilian PV plant. Energy Build. 2015, 83, 248–257. [Google Scholar] [CrossRef]
  109. Marini, D.; He, M.; Buswell, R.A.; Hopfe, C.J.; Crawley, D. Modelling and calibration of a domestic building using high-resolution monitoring data. In Proceedings of the 2016 3rd Conference of IPBSA-England: Building Simulation and Optimization (BSO2016), Newcastle, UK, 12–14 September 2016. [Google Scholar]
  110. Bertagnolio, S.; Lemort, V.; Andre, P. Simulation Assisted Audit & Evidence Based Calibration Methodology; IEA: São Paulo, Brazil, 2010; Volume 53. [Google Scholar]
  111. Badescu, V.; Gueymard, C.A.; Cheval, S.; Oprea, C.; Baciu, M.; Dumitrescu, A.; Iacobescu, F.; Milos, I.; Rada, C. Computing global and diffuse solar hourly irradiation on clear sky. Review and testing of 54 models. Renew. Sustain. Energy Rev. 2012, 16, 1636–1656. [Google Scholar] [CrossRef]
  112. Wang, L.; Gong, W.; Hu, B.; Lin, A.; Li, H.; Zou, L. Modeling and analysis of the spatiotemporal variations of photosynthetically active radiation in China during 1961–2012. Renew. Sustain. Energy Rev. 2015, 49, 1019–1032. [Google Scholar] [CrossRef]
  113. Hu, B.; Liu, H.; Wang, Y. Investigation of the variability of photosynthetically active radiation in the Tibetan Plateau, China. Renew. Sustain. Energy Rev. 2016, 55, 240–248. [Google Scholar] [CrossRef]
  114. Raftery, P. Calibrated Whole Building Energy Simulation: An Evidence-Based Methodology. Ph.D. Thesis, College of Engineering and Informatics, National University of Ireland, Dublin, Ireland, 2011. [Google Scholar]
Figure 1. Fragment of Table 5-2 (ASHRAE Guideline 14 [9]).
Figure 1. Fragment of Table 5-2 (ASHRAE Guideline 14 [9]).
Energies 10 01587 g001
Figure 2. Fragment of Table 5-3 (ASHRAE Guideline 14 [9]) in which the M B E abbreviation is first used instead of N M B E .
Figure 2. Fragment of Table 5-3 (ASHRAE Guideline 14 [9]) in which the M B E abbreviation is first used instead of N M B E .
Energies 10 01587 g002
Table 1. Calibration criteria of the Federal Energy Management Program (FEMP), ASHRAE Guideline 14 and IPMVP.
Table 1. Calibration criteria of the Federal Energy Management Program (FEMP), ASHRAE Guideline 14 and IPMVP.
Data TypeIndexFEMP Criteria [11,12]ASHRAE Guideline 14 [8,9]IPMVP [16]
Calibration criteria
Monthly criteria % N M B E ±5±5±20
C V ( R M S E ) 1515-
Hourly criteria % N M B E ±10±10±5
C V ( R M S E ) 303020
Model recommendation
R 2 ->0.75>0.75
Table 2. Importance of the use of the N M B E uncertainty index to perform comparisons between models for calibration purposes.
Table 2. Importance of the use of the N M B E uncertainty index to perform comparisons between models for calibration purposes.
MonthMeasured
(J)
Simulated
(J)
Difference
(J)
Measured
(kWh)
Simulated
(kWh)
Difference
(kWh)
January9.6506 × 1099.6606 × 109−1.00 × 1072680.722683.50−2.78
February6.704 × 1096.714 × 109−1.00 × 1071862.231865.01−2.78
March4.7898 × 1094.7998 × 109−1.00 × 1071330.511333.28−2.78
April2.5153 × 1092.5253 × 109−1.00 × 107698.68701.46−2.78
May5.4644 × 1085.5644 × 108−1.00 × 107151.79154.57−2.78
June2.5425 × 1082.6425 × 108−1.00 × 10770.6273.40−2.78
July1.5667 × 1091.5767 × 109−1.00 × 107435.20437.98−2.78
August1.1871 × 1091.1971 × 109−1.00 × 107329.75332.53−2.78
September2.7977 × 1082.8977 × 108−1.00 × 10777.7180.49−2.78
October1.1759 × 1091.1859 × 109−1.00 × 107326.64329.42−2.78
November6.0358 × 1096.0458 × 109−1.00 × 1071676.611679.39−2.78
December9.3627 × 1099.3727 × 109−1.00 × 1072600.762603.54−2.78
M B E −1.00 × 107 M B E −2.78
Mean (Measured)3.6724 × 109Mean (Measured)1020.10
N M B E −0.27% N M B E −0.27%
Table 3. Articles and other documents with errors identified.
Table 3. Articles and other documents with errors identified.
TitleYearJournal/Report/ThesisRef.Statistical Indices UsedErrorSourceAbout
Evaluation of overall thermal transfer value (OTTV) for commercial buildings constructed with green roof2013Applied energy[59]MBE, RMSE1-Building energy simulations
A model calibration framework for simultaneous multi-level building energy simulation2015Applied Energy[60]MBE, CVRMSE1ASHRAE 14-2002, IPMVP, FEMPBuilding energy calibration
Expanding Inter-Building Effect modeling to examine primary energy for lighting2014Energy and Buildings[61]MBE, RMSE1ASHRAE 14-2002Building energy simulation
Estimating the impact of climate change and local operational procedures on the energy use in several supermarkets throughout Great Britain2016Energy and Buildings[62]MBE, CVRMSE, R 2 1[63]Energy consumption
Assessment of SUNY version 3 global horizontal and direct normal solar irradiance in Canada2012Energy Procedia[64]MBE, RMSE1-Solar radiation calculations
Generation of typical meteorological years using genetic algorithm for different energy systems2016Renewable Energy[65]MBE, RMSE1-Weather calculations
Simple model for estimating global solar radiation1985Solar and Wind Technology[66]RMSE, MBE1-Solar radiation calculations
A transdisciplinary approach on the energy efficient retrofitting of a historic building in the Aegean Region of Turkey2015Energy and Buildings[67]MBE, RMSE, CVRMSE2ASHRAE 14-2002Building energy retrofitting
Ongoing commissioning of water-cooled electric chillers using benchmarking models2012Applied energy[68]MBE, CVRMSE, RMSE, R 2 3ASHRAE 14-2002Commissioning of electric chillers
Operation and control strategies for multi-storey double skin facades during the heating season2012Energy and Buildings[69]MBE, RMSE, R 2 3ASHRAE 14-2002Energy strategies
Modeling hourly and daily fractions of UV, PAR and NIR to global solar radiation under various sky conditions at Botucatu, Brazil2009Applied Energy[70]MBE, RMSE4-Solar radiation calculations
The role of clouds in improving the regression model for hourly values of diffuse solar radiation2012Applied Energy[71]MBE, AIC, R 2 , RMSE4-Solar radiation in relation with clouds
Human-based energy retrofits in residential buildings: A cost-effective alternative to traditional physical strategies2014Applied Energy[72]MBE, CVRMSE4ASHRAE 14-2002
Historical buildings: Multidisciplinary approach to structural/energy diagnosis and performance assessment2015Applied Energy[73]MBE, CVRMSE4ASHRAE 14-2002Structural/energy diagnosis of a building
Development of a model predictive control framework through real-time building energy management system data2015Applied Energy[74]MBE, CVRMSE4ASHRAE 14-2002Model predictive control
Why is the reliability of building simulation limited as a tool for evaluating energy conservation measures?2015Applied Energy[75]MBE, CVRMSE4ASHRAE 14-2002, IPMVP, FEMPLimits of energy simulation
An EnergyPlus whole building energy model calibration method for office buildings using occupant behavior data mining and empirical data2014Carnegie Mellon University, ASHRAE/IBPSA-USA[76]MBE, CVRMSE4ASHRAE 14-2002Building energy calibration
Development and validation of a Radiance model for a translucent panel2006Energy and Buildings[77]MBE, RMSE4-Daylight study
Heating system performance estimation using optimization tool and BEMS data2008Energy and Buildings[78]MBE, CVRMSE4ASHRAE 14-2002Heating systems
Calibrating whole building energy models: An evidence-based methodology2011Energy and Buildings[79]MBE, CVRMSE4ASHRAE 14-2002, IPMVP, FEMPBuilding energy calibration
A comprehensive analysis of the impact of occupancy parameters in energy simulation of office buildings2012Energy and Buildings[80]MBE4FEMP 3.0, ASHRAE 14-2007 (error, is 2002)Occupancy behavior and energy consumption
Analysis of building energy consumption parameters and energy savings measurement and verification by applying eQUEST software2013Energy and Buildings[81]MBE, RMSE4-Building Energy Consumption
On-site monitoring and dynamic simulation of a low energy house heated by a pellet boiler2016Energy and Buildings[82]RMSD, MBE, CVRMSD, f cost function4ASHRAE 14-2002Building energy calibration
Modelling of a Multi-purpose Commercial Building for Demand Response Analysis2015Energy Procedia[83]MBE4-Demand Response
Calibration of Building Energy Simulation Models Based on Optimization: A Case Study2015Energy Procedia[84]MBE, CVRMSE4ASHRAE 14-2002Building energy calibration
Experimental and numerical study on thermal performance of new cool clay tiles in residential buildings in Europe2015Energy Procedia[85]MBE, CVRMSE4ASHRAE 14-2002Building energy simulation
A review of the CIE general sky classification approaches2014Renewable and Sustainable Energy Reviews[86]MBE, CVRMSE4-Sky classification
A Procedure to Perform Multi-Objective Optimization for Sustainable Design of Buildings2016Energies[87]MBE, CVRMSE5-Optimization of buildings
The all-sky meteorological radiation model: proposed improvements2006Applied energy[88]MBE, RMSE, R 2 6-Solar radiation calculations
Model calibration for building energy efficiency simulation2014Applied energy[89]MBE, CVRMSE7ASHRAE 14-2002Building energy calibration
Evaluation of “Autotune” calibration against manual calibration of building energy models2016Applied energy[90]MBE, CVRMSE7ASHRAE 14-2002, IPMVP, FEMPBuilding energy calibration
Office building cooling load reduction using thermal analysis method–a case study2016Applied Energy[91]MBE, CVRMSE7ASHRAE 14-2002Building energy simulation
Methodology of the cost-optimality for improving the indoor thermal environment during the warm season. Presentation of the method and application to a new multi-storey building in Berlin2017Applied Energy[92]MBE7FEMP 3.0Improve indoor conditions
ESL-TR-94/07-01, Instructions for “Great Energy Predictor Shootout II: Measuring Retrofit Energy Savings”1994ASHRAE[93]MBE, CVRMSE7--
Guideline 14-2002, Measurement of Energy and Demand Savings2012ASHRAE[9]NMBE, MBE, CVRMSE7-ASHRAE Guideline
ANN-Based Prediction and Optimization of Cooling System in Hotel Rooms2015Energies[94]MBE7-Building energy simulation
Study on Auto-DR and pre-cooling of commercial buildings with thermal mass in California2010Energy and Buildings[95]MBE, CVRMSE, RMSE7ASHRAE 14-2002Building energy simulation
Calibrating whole building energy models: Detailed case study using hourly measured data2011Energy and Buildings[96]MBE, CVRMSE7ASHRAE 14-2002Building energy calibration
Optimization of an envelope retrofit strategy for an existing office building2012Energy and Buildings[97]MBE, RMSE7ASHRAE 14-2002, IPMVP, FEMPBuilding energy optimization
Building hourly thermal load prediction using an indexed ARX model2012Energy and Buildings[98]MBE, CV, EEP, σ 7-Building thermal load prediction
Analysis of the impact of using synthetic data correlated with measured data on the calibrated as-built simulation of a commercial building2013Energy and Buildings[99]MBE, CVRMSE7ASHRAE 14-2002Building energy calibration
Calibrated simulation of a public library HVAC system with a ground-source heat pump and a radiant floor using TRNSYS and GenOpt2015Energy and Buildings[100]MBE, CVRMSE7ASHRAE 14-2002Building energy calibration
Energy saving potential through Energy Conservation Building Code and advance energy efficiency measures in hotel buildings of Jaipur City, India2015Energy and Buildings[90]MBE, CVRMSE7IPMVPEnergy efficiency
A simplified PEM fuel cell model for building cogeneration applications2015Energy and Buildings[101]MBE, CVRMSE7-PEM fuel cell model
Application of a simplified thermal network model for real-time thermal load estimation2015Energy and Buildings[102]MBE, CVRMSE7FEMP 3.0Building energy simulation
Building model calibration using energy and environmental data2015Energy and Buildings[103]MBE, CVRMSE7ASHRAE 14-2002Building energy calibration
Analysis of energy efficiency retrofit scheme for hotel buildings using eQuest software: A case study from Tianjin, China2015Energy and Buildings[104]MBE, CVRMSE7ASHRAE 14-2002, IPMVP, FEMPBuilding energy retrofitting
A process for developing deep energy retrofit strategies for single-family housing typologies: Three Toronto case studies2016Energy and Buildings[105]NMBE, CVRMSE7ASHRAE 14-2002Energy Retrofits
Development of a new multi-stage building energy model calibration methodology and validation in a public library2017Energy and Buildings[106]MBE, CVRMSE7ASHRAE 14-2002Building calibration
A combination of Heliosat-1 and Heliosat-2 methods for deriving solar radiation from satellite images2014Energy Procedia[107]MAE, MAE(%), MBE, MBE(%), RMSE, RMSE(%)7-Solar radiation calculations
Development of models for on-line diagnostic and energy assessment analysis of PV power plants: the study case of 1 MW Sicilian PV plant2015Energy Procedia[108]MBE, RMSE, nRMSE7-PV energy analysis
Modelling and calibration of a domestic building using high-resolution monitoring data2016IBPSA[109]MBE, CVRMSE7ASHRAE 14-2002Building calibration
Simulation assisted audit & Evidence based calibration methodology2010IEA-ECBCS Annex 53[110]MBE, RMSE, CVRMSE7ASHRAE 14-2002, IPMVP, FEMPEnergy calibration methodology
Computing global and diffuse solar hourly irradiation on clear sky. Review and testing of 54 models2012Renewable and Sustainable Energy Reviews[111]MBE, RMSE7-Solar radiation calculations
A review of methods to match building energy simulation models to measured data2014Renewable and sustainable energy reviews[63]MBE7ASHRAE 14-2002, IPMVP, FEMPBuilding Calibration
Modeling and analysis of the spatiotemporal variations of photosynthetically active radiation in China during 1961–20122015Renewable and Sustainable Energy Reviews[112]RE, MBE, MABE, RMSE, R 2 7-Photosynthetically active radiation
Investigation of the variability of photosynthetically active radiation in the Tibetan Plateau, China2016Renewable and Sustainable Energy Reviews[113]MBE, MABE, RMSE7-Solar radiation
Calibrated whole building energy simulation: An evidence-based methodology2011Thesis[114]MBE, CVRMSE7ASHRAE 14-2002, IPMVP, FEMPBuilding calibration

Share and Cite

MDPI and ACS Style

Ruiz, G.R.; Bandera, C.F. Validation of Calibrated Energy Models: Common Errors. Energies 2017, 10, 1587. https://doi.org/10.3390/en10101587

AMA Style

Ruiz GR, Bandera CF. Validation of Calibrated Energy Models: Common Errors. Energies. 2017; 10(10):1587. https://doi.org/10.3390/en10101587

Chicago/Turabian Style

Ruiz, Germán Ramos, and Carlos Fernández Bandera. 2017. "Validation of Calibrated Energy Models: Common Errors" Energies 10, no. 10: 1587. https://doi.org/10.3390/en10101587

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop