An Assessment of the Key Performance Indicators (KPIs) of Energy Efficient Retrofits to Existing Residential Buildings
Round 1
Reviewer 1 Report
This paper carried out a comprehensive literature review and identified several KPIs to be used in evaluating energy efficient retrofits. The review part was organized in a rational way and the contents presented was solid and convincing. However, there is no research method proposed except the case studies. It is strongly suggested from the reviewer to summarize and propose a research method and present the innovation of this method. Case studies could be used to validate the proposed method. This is important to be published in a research journal.
Author Response
Response to Reviewer 1 Comments
Point 1: This paper carried out a comprehensive literature review and identified several KPIs to be used in evaluating energy efficient retrofits. The review part was organized in a rational way and the contents presented was solid and convincing.
Response 1: The authors thank the reviewer for their feedback on the literature review.
Point 2: However, there is no research method proposed except the case studies. It is strongly suggested from the reviewer to summarize and propose a research method and present the innovation of this method. Case studies could be used to validate the proposed method. This is important to be published in a research journal.
Response 2: The authors agree with this feedback. In light of such, the paper has been re-structured so that the methodology is presented in Section 3, followed by the case study chapter in Section 4. The methodology section has also been revised, so that case study specific information (previously detailed in the methodology section) has been removed and added to the case study section where it is most relevant. In doing this, the method proposed for the evaluation of each KPI is clearer, and more applicable to other research, while the case study validates how the proposed methods can be used in the Irish context.
Any case specific information relating to the methods proposed are now detailed in Section 4. This includes case specific information for the calculation of (i) thermal comfort (see lines 518-531), (ii) fuel poverty alleviations (see lines 532-536), (iii) annual energy demand savings (see lines 537-554), (iv) annual CO2 emissions savings (see lines 556-558) and (v) pre- and post-retrofit EPCs (see lines 559-567). Specific information on how the calculation of annual electricity energy usage was conducted (especially in cases where electricity was used for space heating, as well as lighting and appliances) are detailed under each specific case study in Sections 4.1-4.5. The reason for this approach is to make it easier for reader to understand these calculations for each case study, in light of the differences in how electricity was consumed across the case studies, pre- and post-retrofit.
The innovativeness of the proposed integrated framework is highlighted in Section 2.4, which states “As is evident from the literature review presented in Section 2.1-2.3, there are a lack of studies which evaluate energy retrofit projects using economic, social, and environmental KPIs. Typically, existing studies evaluate economic and environmental KPIs, while neglecting social KPIs. Existing studies also tend to focus on evaluating one or two categories of KPIs, but very rarely, evaluate several categories of KPIs, within an integrated framework. Thus, an integrated framework for evaluating the wider benefits of dwelling retrofits, consisting of economic, social, and environmental KPIs (Table 2), is proposed and assessed in this study. The methods used to evaluate these KPIs are detailed in Section 3. This framework is applied in a pre-and post-retrofit assessment of five case study dwellings located in Ireland, to reveal and validate its usefulness in demonstrating the wider benefits of retrofitting to householders. Further detail of these case study dwellings is provided in Section 4.”
Reviewer 2 Report
The thematic and research area of the paper is interesting, due to the lack of knowledge of these aspects.
Nevertheless, I would suggest corrections and additions as follows:
I would recommend combining a case study chapter with a method chapter into a methodology chapter.
The selected subsection titled Discussion is needed, so the discussion chapter should be a separate chapter. It should be also extended. This section needs in-depth analysis in the context of the current state of the art on energy efficient retrofits; resilience and sustainability.
In my opinion, the submitted paper carefully develops the main research problem. The analysis of case studies results in fruitful knowledge, inc. European requirements.
Author Response
Response to Reviewer 2 Comments
Point 1: I would recommend combining a case study chapter with a method chapter into a methodology chapter.
Response 1: The authors thank the reviewer for this helpful feedback. The paper has been restructured, with the methodology presented in Section 3 of the paper, followed by the case studies in Section 4. See response to Reviewer 1, Point 2 for further details.
Point 2: The selected subsection titled Discussion is needed, so the discussion chapter should be a separate chapter. It should also be extended. This section needs in-depth analysis in the context of the current state of the art on energy efficient retrofits; resilience and sustainability.
Response 2: The discussion has been made an individual chapter, and extended as suggested. This section now provides a discussion of the findings of the literature review on the current state of existing literature relating to the evaluation of energy retrofit projects and the use of KPIs in such, as well as the importance of demonstrating the wider benefits of retrofitting in the context of incentivising future retrofit uptake. A discussion is provided on particularly interesting findings on the performance of state-of-the-art heat pump and PV systems installed as part of the retrofit works to three case study dwellings and state-of-the-art heating controls installed in two of the case study dwellings. In addition, a discussion of the findings relating to householder experiences of using these state-of-the-art measures is also provided. These findings have been compared with existing literature, with a discussion of the implications of such findings in light of policy targets relating to the adoption of these technologies in Ireland, including the Irish government’s Climate Action Plan targets of 400,000 heat pumps installed in existing dwellings by 2030. Thus, the usefulness of an integrated framework (such as that of this paper) for future design of these technologies, retrofit packages, and policy support mechanisms is also discussed. A discussion of the limitations have also been added to this section (Section 6.1). (Refer to Reviewer 3, Point 7 for more information).
Author Response File: 
Author Response.docx
Reviewer 3 Report
The article provides an interesting read on the KPI of energy-efficient retrofitting of existing residential buildings with multiple case studies. There are a few points to address before the article will be published.
1) The structure of the article must be improved for the purpose of readability and consistency. Hence, the case studies in section 3 should be after the methods in section 4.
2) The methods in section 4 must indicate/express the case study protocol. Are there any embedded cases? The authors should consider the following:
https://espace.curtin.edu.au/handle/20.500.11937/45440
https://www.scienceopen.com/hosted-document?doi=10.14236/ewic/EASE2008.5
3) In the methods, line 409, the authors stated:
"Pre- and post-retrofit, face-to-face semi-structured surveys were conducted with an adult (aged 18 years or older) in each household. Information was gathered on the householders’ demographic profiles, their attitudes towards energy use and conservation, quality of life and the environment, which items they viewed to be necessities or luxuries, their 412 energy-related practices, and their thermal satisfaction within their homes".
The authors need to provide more details on the interviewees in terms of profile, and findings.
4) It will be important for the authors to discuss the sampling, sample sizes and add some justifications for the case study and the interviews.
5) The findings of the interviews must be presented and triangulated with the case study.
6) Separate the discussion from the conclusion of the study in section 6. Provide a single section to discuss the implications of the findings for policy, practice and contribution to knowledge.
7) What are the limitations of the study.
8) The reference list is not in accordance with the journal requirements. Improve the presentation. The style of referencing is ACS.
Useful link for converting the references:
https://app.bibguru.com/?style=american-chemical-society
9) Authors must provide another round of proofreading to eliminate any syntax or grammatical errors.
Author Response
Response to Reviewer 3 Comments
Point 1: The structure of the article must be improved for the purpose of readability and consistency. Hence the case studies in section 3 should be after the methods in section 4.
Response 1: The authors thank the reviewer for this helpful feedback. This comment has been addressed. Please refer to response given to Reviewer 1, Point 2 for further details.
Point 2: The methods in section 4 must indicate / express the case study protocol. Are there any embedded cases? The authors should consider the following:
https://espace.curtin.edu.au/handle/20.500.11937/45440
https://www.scienceopen.com/hosted-document?doi=10.14236/ewic/EASE2008.5
Response 2: The authors thank the reviewers for this feedback, and for suggesting sources to assist with such. The case study protocol has been outlined, and is included in Appendix A of the paper. It is added as an Appendix as opposed to section 4, in order to save the methodology section becoming overly long for the reader.
Point 3: In the methods, line 409, the authors stated:
“Pre- and post-retrofit, face-to-face semi-structured surveys were conducted with an adult (aged 18 years or older) in each household. Information was gathered on the householders’ demographic profiles, their attitudes towards energy use and conservation, quality of life and the environment, which items they viewed to be necessities or luxuries, their energy-related practices, and their thermal satisfaction within their homes.” The authors need to provide more details on the interviewees in terms of profile, and findings.
Response 3: The authors acknowledge that a large data set was gathered through these semi-structured surveys. However, analysis of all of the available study was outside the scope of the study. The authors now acknowledge this in the paper (see lines 375-389), and refer the reader to another study by two of the authors, that provide further analysis of this data set. “Analysis of the remaining data was outside of the scope of this study. Further details on the other type of data collected as part of the semi-structured surveys are given in with the data used to examine the impact retrofitting has on the energy cultures and the sustainable-related outcomes among inhabitants of social housing units in Ireland [91].”
Point 4: It will be important for the authors to discuss the sampling, sample sizes, and add some justifications for the case study and the interviews.
Response 4: This information is now provided in Appendix A of the revised paper with the case study protocol information. In addition, the authors discuss and justify this particular sample in the limitations of the study. See lines 919-923.
“It is also acknowledged that a small sample of just five case study dwellings was used in this study. The case studies used in this paper were part of a wider monitoring study, consisting of a total of 13 dwellings. However, due to the loss of data collected by the data loggers in three of the eight dwellings that invested in a retrofit, five case study dwellings were selected for use in this study.”
Point 5: The findings of the interviews must be presented and triangulated with the case study.
Response 5: As mentioned in Response 3, only particular data from the surveys was used for the purpose of this study and the analysis of the KPIs. This includes data on (i) householder demographics and socioeconomics, (ii) householder’s thermal satisfaction within their homes, (iii) household self-reported energy use, and (iv) satisfaction with various aspects of the retrofit. Householder demographics and socio-economic information are detailed in Section 4.1-4.5 for each case study dwelling. Findings from the surveys relating to householder thermal satisfaction within their homes are highlighted in Table 3, as well as within the findings of each case study dwelling in Sections 5.1-5.5. Select quotes from the surveys are used. The findings relating to self-reported energy use are used in the calculation of annual energy demand savings and annual CO2 emission savings, and the findings from the surveys relating to householder satisfaction are highlighted in Table 3, while select quotes from the surveys are presented in Sections 5.1-5.5.
Point 6: Separate the discussion from the conclusion of the study in section 6. Provide a single section to discuss the implications of the findings for policy, practice, and contribution to knowledge.
Response 6: The discussion is now presented as a separate section from the conclusions. The conclusion section now presents the key findings and the implications for policy, practice, and knowledge. See response to Reviewer 2, Point 2 for additional information.
Point 7: What are the limitations of the study?
Response 7: The limitations are now presented in the discussion section (and acknowledged as necessary elsewhere in the paper). The limitations are outlined below:
- It is acknowledged that the differences in the case study dwellings presented makes it difficult for a cross-comparison of the benefits received following each retrofit
- While this study has demonstrated the usefulness of an integrated framework of KPIs in demonstrating the economic, social, and environmental benefits of retrofitting, some limitations should be acknowledged. Firstly, even though the KPIs representing each core category assessed are justified in their inclusion, they are certainly non-exhaustive, with other KPIs in each category being discussed in the literature. They are considered to be the minimum captured for demonstrating the wider benefits of retrofitting to homeowners and encouraging homeowners to invest in building retrofits. As retrofit projects are complex and can have other benefits that may not be captured using the selected set of KPIs, other KPIs should be added if required. In addition, it should be noted that while the KPIs assessed, and the methods used for their assessment, resulted in useful knowledge on the wider benefits of retrofitting, it is not suggested that these methods be considered a standard methodology for the assessment of these KPIs in the future. There is a multitude of ways to assess these KPIs, and further, critical analysis of best practice methodologies for doing so is needed. This is an interesting direction for future research, particularly, for developing standardised integrated frameworks that can be applied to large sample sets. Other researchers agree on the importance of developing standardised methods for the assessment of such KPIs [33].
- It is also acknowledged that a small sample of just five case study dwellings was used in this study. The case studies used in this paper were part of a wider monitoring study, consisting of a total of 13 dwellings. However, due to the loss of data collected by the data loggers in three of the eight dwellings that invested in a retrofit, five case study dwellings were selected for use in this study.
Point 8: The reference list is not in accordance with the journal requirements. The style of referencing is ACS.
Useful link for converting the references:
https://app.bibguru.com/?style=american-chemical-society
Response 8: The authors thank the reviewer for providing this useful tool. The reference list has now been updated to the appropriate style.
Point 9: Authors must provide another round of proofreading to eliminate any syntax or grammatical errors.
Response 9: A further round of proof-reading has been completed.
Author Response File: Author Response.docx
Round 2
Reviewer 1 Report
The paper has been revised significantly and meet the expectations from the reviewer.
Author Response
Thank you for completing this review and we are happy that your expectations.
Reviewer 3 Report
The manuscript has been revised extensively and my review comments have been addressed. However, there are some minor issues to resolve before publication.
1) Line 392, section 3.1 stated that an interview was used as part of the methodology. However, the authors' responses indicated that the interview was not analysed or used in this study. Therefore, it must be removed if it was not analysed in this study. There is no reason for including a data collection instrument that was not applied in the manuscript. This study can just focus on the case study alone without mentioning the interview data collection which is redundant.
2) The content of Appendix A should be brought into the main manuscript. This should be under the methods in section 3.1. If the authors feel that this should be in another section apart from 3.1, then it can be added just before the case study. The content provided in the appendix is very important and must not be left out of the methods section.
Author Response
Responses to Reviewer 3
The manuscript has been revised extensively and my review comments have been addressed.
Response 1: Thank you for taking the time to review our revised manuscript. We are happy that you are satisfied that your review comments have been addressed.
However, there are some minor issues to resolve before publication.
1) Line 392, section 3.1 stated that an interview was used as part of the methodology. However, the authors' responses indicated that the interview was not analysed or used in this study. Therefore, it must be removed if it was not analysed in this study. There is no reason for including a data collection instrument that was not applied in the manuscript. This study can just focus on the case study alone without mentioning the interview data collection which is redundant.
Response 2: The level of householder satisfaction with various aspects of the retrofit was also gathered through the interviews and the results of which are included in Section 5. In Section 3.1 we note that in the interviews we gathered additional information that wasn’t included in the analysis presented in this paper, such as the householders’ demographic profiles, their attitudes towards energy use and conservation, quality of life and the environment, which items they viewed to be necessities or luxuries, their energy-related practices, and their thermal satisfaction within their homes. We feel it’s important for the reader to be aware that the set of questions used in this paper form part of a larger questionnaire and refer the reader to a previous publication involving authors of this paper [Ref 91]. Thus, we wish to leave Section 3.1 as it is currently written.
2) The content of Appendix A should be brought into the main manuscript. This should be under the methods in section 3.1. If the authors feel that this should be in another section apart from 3.1, then it can be added just before the case study. The content provided in the appendix is very important and must not be left out of the methods section.
Response 3: There is already a lot of information packed into the main body of the paper. Thus, we feel that the content in Appendix A is best located in an appendix. However, to help the reader identify that this information is available, we have added in a new sentence into Section 3.1 and Section 4.
In the opening paragraph of Section 3.1 and Section 4, we have added “Details of the procedure followed to recruit potential research participants is given in Appendix A.”
Author Response File: Author Response.docx
