Monitoring the Sustainability of Building Renovation Projects—A Tailored Key Performance Indicator Repository

Abstract

This study aims to assist in the identification of suitable key performance indicators (KPIs) that can be used to assess the sustainability performance of buildings given their transition into zero-carbon, resource-efficient, and resilient structures. To that end, a four-step methodology is proposed in this work; the first step includes the development of a KPI repository, which builds upon commonly accepted targets derived from the needs of different stakeholders as well as targets imposed by external factors. The second step refers to the expansion of the initial KPI repository, capitalizing on information from the literature. The third step includes the refinement of the repository based on predefined criteria (relevance, availability, measurability, reliability, and familiarity) and tailored feedback from key stakeholders. The final step concerns the development of KPI cards, which include all the necessary information for understanding and estimating the KPIs included in the final repository. This four-step methodological approach implementation was tested during the EU-funded HORIZON project ‘InCUBE’. The implementation of the first two steps of the proposed methodology resulted in a pool of 68 KPIs. Nearly half of these KPIs were extracted from Step 1 to fully support the monitoring of all InCUBE outcomes, while the rest of the KPIs were extracted from highly relevant Horizon frameworks, the B4P partnership, Level(S) framework, publications, and ETSI standards (Step 2). The implementation of Step 3 resulted in a shortlisted KPI pool which eventually defined the final InCUBE KPI repository, including 31 KPIs. To help with the coordination of the data gathering process and a shared understanding of the sort of information to be monitored among various stakeholders, selected KPI cards (Step 4) are presented.

1. Introduction

Given that the building sector has one of the highest carbon footprints, decarbonizing it by 2050 will be essential in achieving the Paris Agreement’s [1] goal of reducing emissions. The sector’s operations rebounded by 2% more than the all-time high of 2019 after the pandemic-related outlier of emission patterns in 2021, despite an increase in energy efficiency investments of roughly 16% [2]. The application of renewable energy sources (RES) technologies in buildings remains modest but the increase in fossil fuel cost made such investments more attractive alongside green building certification improvements, which saw a 19% increase compared to 2020. In the midst of those urgent decarbonization challenges, the global gross floor area has increased from 218 billion m² in 2015 to 242 billion m² in 2021 [2]. The construction industry, however, has long been thought to be underperforming. For instance, construction is distinguished by having much lower rates of productivity than other economic sectors (1% versus a global economy average of 2.8 percent [3]) and low levels of user satisfaction; on average, construction projects experience time delays of 20% and cost overruns of 80% [4]. Today’s construction practices typically take a project-based approach, which limits consistency and replicability [3]. Despite the advantages afforded in the new era of digitization [3], projects are delivered through a fragmented value chain with misaligned contractual arrangements. This is particularly true in the renovation industry, where a substantial number of subcontractors stem from actors in specialized trades (up to 95% are micro- or SME-enterprises [5]). The COVID-19 crisis brought to light the logistical issues in the construction industry’s supply chain, which caused delays in some building projects since unique products made of cutting-edge materials had to be sent from overseas while manufacturing was halted [6].

However, it is anticipated that future ecosystems for renovation will be more cooperative, technologically sophisticated, and environmentally sound [7]. In terms of sustainability, there is a growing awareness of the environmental impact of buildings, and a push towards more sustainable practices. Renovations will increasingly focus on energy efficiency, integration of renewable energy systems [8], the use of renewable materials, and waste reduction [9]. Technology is transforming the renovation process, from digital design tools to smart building systems that can monitor and control energy use. Augmented and virtual reality can also help homeowners visualize renovation designs and make informed decisions [10]. Collaboration across the value chain will become more important, as builders, architects, designers, and suppliers work together to deliver more sustainable and efficient renovations. This could involve more integrated project delivery methods or the use of online platforms that connect homeowners with skilled tradespeople and contractors. The pandemic has highlighted the importance of healthy and safe homes. Renovations will increasingly focus on an integrated system for improving indoor air quality, natural lighting, and the use of materials that are free of toxins and allergens [11]. With the increasing frequency and severity of extreme weather events, renovations will need to be designed to withstand the impacts of climate change. This could involve measures such as reinforcing roofs, walls, and foundations or incorporating flood-resistant features. Sustainability, technology, cooperation, health and wellbeing, and resilience will all play major roles in the future renovation environment. To meet the changing needs of homeowners and the environment as society continues to place greater emphasis on these values, the renovation sector will need to adapt and develop.

The monitoring and evaluation (M&E) of renovation projects is crucial to ensure that a project is on track and achieving its intended goals [12]. Initially, the establishment of specific goals and objectives is necessary for the effective monitoring and evaluation of sustainable renovation projects. Prior to beginning a renovation project, it is crucial to establish specific, quantifiable, achievable, relevant, and time-bound sustainability goals and objectives that adhere to the SMART (specific, achievable, relevant, time-bound) principles. Project management software, digital dashboards, and mobile apps that track progress and gather data can all be effective tools for monitoring and analysing renovation projects. A comprehensive M&E plan that specifies the metrics, information sources, and processes for monitoring progress and assessing the project’s success is crucial [13]. Regular reporting mechanisms to ensure that progress is being tracked, and data are being collected regularly are also important. This could include weekly or monthly status updates, progress reports, and performance metrics, which could hold project teams accountable for the progress toward goals. This accountability ensures that everyone involved in the renovation project is working towards the same objectives and that the project is meeting stakeholder expectations. Stakeholders’ involvement in the monitoring and evaluation process could include the project team, contractors, funders, and end-users [14]. Finally, documenting lessons learned during the monitoring and evaluation process and systematically using this knowledge to inform future renovation projects could enhance the performance of renovation projects.

As there are a number of repositories with hundreds of used and/or suggested KPIs, (the draft ISO Standard on Smart Cities and Communities (TC 268) contains about 400 city indicators), the main objective of this study is to present a methodology for KPI extraction that guarantees that important variables connected to the evaluation of deep renovation operations as mandated by both internal and external conditions are considered. Limiting the final number of indicators to be utilized for evaluation and ensuring that the final repository reflects the needs of the project is necessary to keep the evaluation process manageable and efficient [15]. Furthermore, a commonly accepted methodology for the selection of the most appropriate KPIs for each case is missing from the renovation value chain at the moment [16]. The application of the proposed methodology leads to a customized KPI repository containing a limited number of KPIs that can be practically measured and reflect important sustainability conceptions and requirements to assess deep renovation projects in general, where the end user can also be a construction company not necessarily participating in EU HORIZON projects. The study uses the EU-funded HORIZON project InCUBE as a test bed to apply and validate its ideas. The overall goal of InCUBE is to industrialize renovation workflows, operating in a circular system, and adopt life-cycle thinking at each point (resource and energy consumption through the manufacturing stages, including the raw materials acquisition stage), reusing what is necessary to transform the EU building stock into affordable microcosms of a more resilient, greener, and digitalized society.

This paper provides a standardized methodology and a measurable framework for assessing and communicating the outcomes and benefits of deep building renovation projects to a wide range of stakeholders, i.e., project owners, construction companies, government agencies, and regulators. The proposed framework helps project owners monitor whether a deep renovation project is meeting energy efficiency targets, budget constraints, and has positive social impact. Construction companies can assess their performance in executing deep renovation projects by measuring factors such as energy savings achieved, time needed on-site for renovation works, customer satisfaction, change in property value, and payback time of the implemented interventions or the renovation process as a whole. By monitoring these KPIs, construction companies can identify areas for improvement, optimize their construction processes, and enhance their competitiveness in the market. Government agencies and regulators can also benefit from the proposed framework by tracking the overall energy savings achieved, carbon emissions reduction, and compliance with energy efficiency standards. The proposed methodology establishes clear criteria and metrics, which stakeholders can use to assess and compare deep renovation projects consistently. This ensures that all parties involved have a shared understanding of the project’s objectives and expected outcomes. Finally, it provides stakeholders with reliable information for informed decision making helping to mitigate risks and optimize resource allocation.

The rest of the study Is structured as follows. In Section 2, some key sources, ranging from EU strategic plans and initiatives to scientific publications and standards, are provided, where one can find KPIs for construction and renovation projects. In Section 3, the methodology is presented, including specific guidance on how to apply every step. In Section 4, this methodology is applied to the renovation projects included in InCUBE, leading to a final KPI repository that will be applied to monitor the sustainability of InCUBE renovation activities. The paper concludes in Section 5, where important conclusions are summarized.

2. Literature Review–Indicative Sources to Extract KPIs

The European Union’s strategic goals for the construction industry include developing an integrated and competitive building sector, reducing energy consumption and emissions from the building sector, and encouraging sustainable building materials and technology. The lack of comprehensive data on existing buildings, which makes it difficult to assess and measure the potential for improvement, the absence of a unified regulatory framework across all EU member states, and the absence of financial incentives and support for homeowners to improve the energy efficiency of their homes are the main issues relating to the sustainability of renovation actions in residential buildings in the EU. The public should be made more aware of the benefits of renovations, and the construction industry experts should receive greater training and support. The EU has also created KPIs to track the development of its various initiatives and monitor the energy efficiency of buildings in order to better gauge the effectiveness of its efforts.

The Energy Performance of Buildings Directive (EPBD), adopted by the European Union in 2002 [17], introduced the first framework for enhancing energy efficiency and achieving energy neutrality in buildings. With the ultimate goal of lowering carbon emissions from energy use in buildings, this directive set legally binding targets for the energy performance of both new and existing buildings. To lower carbon emissions from the construction industry and increase energy efficiency, the German DGNB System was put into place in 2009 [18]. The framework was created to reduce greenhouse gas emissions in Germany by 80% by the year 2050. A variety of objectives and standards are outlined in the framework for buildings to be certified as sustainable. These goals and standards are divided into six broad areas, including ecological, economic, sociocultural, technical, process, and site quality. They also offer guidelines for energy efficiency retrofits and thorough details on energy-efficient building materials and construction methods, as well as performance indicators to evaluate, prioritize, and manage project progress. LED lighting, insulation, high-performance windows, and effective heating and cooling systems are a few examples of energy-efficient retrofits. The Renovation Wave Framework [19] was launched in May 2020 by the Energy Performance of Buildings Directive (EPBD) of the European Union. The plan by the European Commission to lower energy consumption and carbon emissions from buildings across the EU includes the framework. In addition to guiding extensive rehabilitation projects, it contains a variety of measures targeted at enhancing energy efficiency and establishing energy neutrality in buildings. Numerous KPIs are included in the Renovation Wave Framework to evaluate and monitor the development of deep renovation projects. These KPIs cover the building’s energy efficiency, indoor air quality, the materials utilized, water usage, trash management, and other environmentally friendly elements like solar panels and green roofing. The framework also emphasizes recognizing and tackling energy poverty and offering resources for involving individuals in the repair process.

EU’s ambitious new growth strategy places energy efficiency and renewability at the forefront of sustainable development (reflected in relevant initiatives, strategic documents, and financial frameworks such as the Clean Energy for all Europeans Package [20], the European Green Deal [21], and the 2021–2027 long-term EU budget and Next Generation EU [22]), which can set off the Renovation Wave [19], offering a great opportunity to make renovation a win-win for climate neutrality and equitable economic recovery. Tools for assessing the energy efficiency of buildings include Level(s), Built4People, and the Smart Readiness Indicator.

Built4People (B4P) is a partnership among several European stakeholders, across the climate, energy, and mobility value chain, aiming to make Europe the first carbon-neutral economy and deliver on the European Green Deal. The partnership’s vision is to enable the widespread development in Europe of a high-quality, low-carbon, and energy and resource-efficient built environment. This is proposed to be achieved through the acceleration of people-centric innovations generated by the entirety of the construction value chain and the meaningful communication between the different sectors of the value chain. Specific focus is directed towards the seamless integration of emerging innovative solutions and systems into the built environment, all while maintaining the sustainability and resilience of the buildings, as well as the health and wellbeing of their inhabitants as the primary goal. Specific attention is additionally drawn to cultural heritage buildings and the unique ways they need to be addressed in terms of renovation and decarbonization, along with the rest of the built environment. The Smart Readiness Indicator is a tool for assessing and monitoring the energy performance of buildings, whereas Level(s) offers a certification system for identifying and recognizing sustainable buildings on the market. Both technologies can assist in identifying areas for improvement or retrofitting as they evaluate the building’s energy efficiency, indoor air quality, materials, water usage, and waste management. The Smart Readiness Indicator also assesses the feasibility of deploying smart technologies like solar power, LED lighting, or sophisticated heating control systems.

Level(s) is a unified framework of key sustainability performance indicators for office and residential buildings in the EU and aims to provide an accessible and integrated solution to project design teams, clients, investors, and policymakers for the evaluation of the performance of a building. The proposed indicators assess the four key sustainability axes examined by the framework; environmental performance, occupant health and comfort, life cycle cost and value, and potential risks to future performance. Level(s)’ core objective is to facilitate a better understanding of the sustainability performance of a building by all stakeholders involved to be able to enhance and optimize it. Specific focus areas for the monitoring and evaluation of resource utilization in the framework are energy and water use, building elements and materials, building designs and structures, maintenance plans, and indoor environmental conditions. The Level(s) common framework is based on six macro-objectives which are, namely, greenhouse gas emissions along a building’s life cycle, resource efficient and circular material life cycles, efficient use of water resources, healthy and comfortable spaces, adaptation and resilience to climate change, and optimized life cycle cost and value. Each of the macro-objectives is accompanied by a list of corresponding indicators, 16 in total. The indicators provide a specific and measurable way to ensure that the building’s performance is monitored correctly and has the potential to be optimized.

The concept of SRI was introduced in the revised Energy Performance of Buildings Directive (EPBD) [23] in 2018 to provide a common EU scheme for assessing the smart readiness of buildings. Following regulations [24,25] and technical studies [26] launched the current SRI testing phase, according to which EU countries can implement, optionally for the moment, this rating scheme. The SRI assesses the ability of a building to sense, interpret, communicate, and actively control in an efficient manner the operation of technical building systems optimizing energy efficiency and overall performance, adapting to signals from the grid (energy flexibility), and responding to the needs of the building occupants [27]. The methodology for calculating the SRI is described in detail [26] and summarized in Figure 1. The final SRI rating depends on the examined building’s ability to facilitate “smart-ready” services which are included in a “smart-ready service catalogue”, addressing nine (9) technical domains: (1) heating; (2) domestic hot water; (3) cooling; (4) ventilation; (5) lighting; (6) dynamic building envelope; (7) electricity; (8) electric vehicle charging; and (9) monitoring and control.

Figure 1. SRI methodology (adapted from Apostolopoulos et al., 2022 [28]).

Apart from the various frameworks and tools, there are many international and EU standards available dealing with the concept of sustainability in construction and, by extension, renovation practices, as well as building operation. The two more prominent standardization organizations offering these guidelines and voluntary frameworks for evaluation are the International Standardization Organization (ISO), at an international level, and the European Committee for Standardization (CEN), at a European level. Standards relevant to sustainability in building construction and the assessment of construction work performance from different perspectives (e.g., environmental, social, and economic), as well as standards targeting the building’s performance and environmental impact during its operation phase, have been published by the International Standardization Organization (ISO) in the past years, as presented in Table 1. Regarding CEN-CENELEC standards, standard CEN/TR 17005:2016 [29], in particular, includes suggestions regarding the assessment of the environmental performance of a building, including relevant indicators, and could be of value for the evaluation of the environmental performance of the renovation projects. Additionally, standards EN 16627:2015 [30], EN 16309:2014+A1:2014 [31], and EN 15978:2011 [32] provide calculation methodologies for the evaluation of the economic, social, and environmental performance of the building, respectively, that could offer guidance.

Table 1. ISO standards targeting the building’s performance and environmental impact.

Code	Title
ISO 21931-1:2022	Sustainability in buildings and civil engineering works—framework for methods of assessment of the environmental, social, and economic performance of construction works as a basis for sustainability assessment—Part 1: buildings [33]
ISO 21678:2020	Sustainability in buildings and civil engineering works—indicators and benchmarks—principles, requirements, and guidelines [34]
ISO 20887:2020	Sustainability in buildings and civil engineering works—design for disassembly and adaptability—principles, requirements, and guidance [35]
ISO 15392:2019	Sustainability in buildings and civil engineering works—general principles [36]
ISO 16745-2:2017	Sustainability in buildings and civil engineering works—carbon metric of an existing building during the use stage—Part 2: verification [37]
ISO 21929-1:2011	Sustainability in building construction—sustainability indicators—Part 1: framework for the development of indicators and a core set of indicators for buildings [38]

Finally, the European Telecommunications Standards Institute (ETSI) [39], while mainly focusing on standards relevant to telecommunication systems, processes, and equipment, has also extended its expertise to smart cities, a concept that can provide evaluation criteria for buildings at a district or, potentially, renewable energy community level. ETSI has issued a specification document, GS OEU 019 [40], titled “Operational energy Efficiency for Users (OEU); KPIs for Smart Cities” including key performance indicators for smart cities in the sectors of people, planet, prosperity, governance, and propagation.

The available literature was also reviewed to evaluate studies where KPIs were defined and used to measure the sustainability aspects of renovation projects and/or to define integrated frameworks for the evaluation of such projects through multi-criteria decision making were taken into account. To account for up-to-date information, studies published during the past five years were considered.

Urbinati et al. (2022) [41] reviewed the available literature in the area of sustainability and proposed a “holistic” framework/scorecard of KPIs based on the triple bottom approach for measuring the performance of building retrofit projects. Jafari and Valentin (2018) [42] also considered the three pillars of assessing sustainability in renovation projects, namely economic, environmental, and social performance, and proposed a Sustainable Energy Retrofit decision support system that aims at maximizing the project benefits through multi-objective optimization considering the lifecycle cost of the projects, total air emissions and the occupant’s comfort and satisfaction level (indoor air quality, temperature, humidity, and controllability). Similarly, McGinley et al. (2022) [43] reviewed the available literature on the use of KPIs for the evaluation of energy projects with a particular focus on studies that take into account the perspectives of various stakeholders and proposed an integrated framework for assessing the energy renovation of buildings from an economic, environmental, and social perspective, considering 11 KPIs.

Other studies considered additional and/or different categories, other than the three main dimensions of the triple bottom approach. Toufeili et al. (2019) [44] noted that energy retrofit projects are commonly evaluated based on environmental and economic criteria and proposed a multi-criteria decision making (MCDM) framework (considering LCA aspects) for the assessment of building energy renovation. The method considers 20 KPIs clustered into four main categories: environmental, economic, social, and technical. Ascione et al. (2022) [45] highlighted the role of the building sector in reducing the energy consumption of the European economy and discussed the policies of the various EU countries on the energy upgrade of buildings. The effectiveness of the Italian policy was investigated through the evaluation of building renovation case studies considering energy indicators (“non-renewable primary energy need index” (EPgl, nren)—absolute and percentage difference to the base case, energy label), environmental (CO₂ emissions difference to the base case) and economic (net present value, discounted payback period and ISI index, i.e., “investment cost” to “annual primary non-renewable energy-saving”) indicators.

Terés-Zubiaga et al. (2020) [46] proposed a methodology developed within the framework of the IEA “Annex 75: Cost-Effective Building renovation at District Level Combining Energy Efficiency and Renewables” for the assessment of building renovation at the district level to determine cost-effective solutions. Three main overarching KPIs are considered, namely (i) GHG emissions (CO₂ eq./m² year), (ii) primary energy consumption (kWh/m² year), and (iii) annualised total costs (EUR/m² year). Furthermore, depending on the specific requirements of each project, additional indicators are proposed also taking into account the demand side (cluster of buildings) and the energy supply side, i.e., heating/cooling demand and heating/cooling consumption at the supply side, DHW demand, and DHW consumption, Electricity demand, electricity consumption, final energy demand, final energy consumption, and ratio of RES to the total energy requirements. Concerning the cost analysis, the LCC approach was followed where the initial investment and replacement costs, the energy costs, and the operational and maintenance costs were considered and transformed into annual costs. Finally, Kylili et al. (2016) [16] reviewed studies that used KPIs for the measurement and assessment of the sustainability performance of renovation projects and classified the wide range of indicators identified (149 KPIs in total) into eight categories: (i) economic, (ii) environmental, (iii) social, (iv) technological, (v) time, (vi) quality, (vii) dispute, and (viii) project administration.

A main challenge identified through the literature in the field of renovation assessment refers to the vast number of KPIs proposed for renovation projects that have not undergone any customization or refinement and can result in a great deal of effort with limited benefits in terms of monitoring and quality assessment [15]. Furthermore, the majority of currently available and widely used KPIs primarily evaluate techno-economic and environmental factors, leaving out additional aspects like the social impact of renovations [43]. Additionally, no common framework for the selection and the evaluation of the relevant KPIs for each case was found [16]. Owing to the fragmented nature of the renovation/construction value chain, the indicators typically used reflect one aspect of the renovation at a time, while a holistic approach to the evaluation of the renovation process is missing [47]. This study develops a methodology to create customized assessment frameworks for renovation projects across the EU that incorporates suggested KPIs from ongoing EU initiatives and standards, as well as successfully implemented projects, in order to improve the evaluation process of building renovation projects in the market as well.

3. Methodology

The proposed methodology for extracting KPIs is based on four (4) major consecutive steps, which ensure that significant aspects related to the evaluation of deep renovation activities as imposed by internal (e.g., own goals, stakeholder needs) and external (e.g., strategic plans, initiatives, scientific literature) sources are both considered. These four steps are summarized in Figure 2.

Figure 2. The 4-step methodology proposed.

The proposed methodology was developed by a team comprising postdoctoral researchers with extensive experience in key performance indicators and assessment frameworks and qualified engineers with a strong background in building renovation and energy efficiency.

To examine and validate whether a renovation project was able to fulfil its goals, it is critical to set up and define appropriate KPIs that can facilitate this process and reflect pre-established targets. In this respect, during Step 1, an initial pool of KPIs can be developed, the quantification of which will facilitate the monitoring of all expected outcomes and eventually validate the success of interventions. These targets can derive from personal goals based upon the needs of different stakeholders, including end-users (e.g., final cost of interventions, achieved energy savings) or targets imposed by external factors such as legislation and/or national-regional requirements (e.g., architectural constraints, the time needed to complete the renovation, final energy performance of the building, etc.).

While Step 1 ensures that all necessary parameters for monitoring a project’s deviations from the original goals are considered, there might still be significant aspects that are of high importance for deep renovation projects and should be included in a KPI repository to monitor sustainability in a holistic way, to expand the initial KPI pool developed during Step 1 by capitalizing on well-known and commonly accepted sources. Indicative sources have already been described in Section 2. Depending on the nature of the project (e.g., level of innovation, spatial characteristics, building typology) and the funding source (public or private funds), sources to be utilized to expand the KPI pool should be selected accordingly.

The implementation of Steps 1 and 2 leads to an extended KPI pool. Adopting the majority of these KPIs would make the monitoring process quite overwhelming and nearly impossible to be applied in practice; thus, there is a need to define a clear selection procedure to narrow down the most appropriate KPIs for inclusion into the final repository (Step 3). The following process is suggested which can be applied to any type of project. First, the criteria upon which every KPI of the project’s pool has to be evaluated need to be defined. We recommend building upon five (5) selection criteria initially proposed by the CIVITAS framework and have subsequently been adopted by several HORIZON projects such as CITYkeys [48], RINNO [49], POCITYF [50], and RESPONSE [51] to shortlist KPIs. These criteria are universal (can be applied to any type of project) and are described below in further detail.

• Criterion #1: Relevance: The specific criterion refers to the importance a KPI has in the evaluation process. In other words, the final KPIs to be included in the repository should serve as much as possible the operational objectives of the project. KPIs should be selected and defined in such a way that the implementation of renovation activities provides a clear signal in the change in the indicator value. KPIs that are influenced by other factors not related to the implementation of the project are not suited. KPIs that provide an ambiguous signal (if there is doubt on the interpretation of, e.g., an increase in the indicator value) are equally not suited.
• Criterion #2: Availability: The specific criterion refers to the availability of data required to quantify a KPI. Data for measuring the indicator should be easily available (limited time and effort needed). KPIs that, while being of interest, cannot be realistically estimated during the project lifetime should be avoided. KPIs should be based, if possible, on data that either: (a) are available from the technology providers or other stakeholders involved in the use case that is being evaluated; (b) can easily be compiled from public sources and open-data repositories; and/or (c) can easily be gathered from interviews, questionnaires, maps, or digital tools. KPIs that require, for instance, extensive interviews with occupants will receive a lower score as the large amounts of data needed are too expensive to gather. The same holds for KPIs that require extensive recalculations and additional data, such as footprint indicators, and some financial indicators.
• Criterion #3: Measurability: The specific criterion refers to the capability of a KPI to be measured, preferably as objectively as possible. It is also important for a KPI to reflect the changes in the measured quantities as quickly as possible so that corrective measures can take place in time and ensure the project’s success. The utilization of KPIs that are of qualitative or semiqualitative nature (e.g., are assessed with the utilization of Likert scales) should be avoided. However, this might not be feasible, especially when, for instance, social KPIs need to be included in the repository.
• Criterion #4: Reliability: The specific criterion refers to the clarity of the definition of a KPI (and the calculation method). The definition and the calculation method of the KPIs should be clear and not open to different interpretations and include parameters of data collection that can affect the quality of the measurements, like spatial and temporal levels. This criterion can be enhanced through the KPI cards (see next step) which will summarize key relevant info.
• Criterion #5: Familiarity: The specific criterion refers to the easiness of comprehension regarding the issue a KPI is addressing. KPIs should be easily understood by users, non-experts if possible. One should rely on KPIs from existing indicator-based frameworks that generally comply with this requirement; however, for several cases the KPI’s definition may not be clear, especially for non-experts.

Once the selection of criteria has been finalized, the evaluation of all KPIs included in the extensive pool, using the predefined criteria, is performed. Each KPI can be evaluated through a 3-point scoring system per criterion; 0 points: the KPI does not satisfy this criterion adequately; 1 point: the KPI satisfies this criterion sufficiently; 2 points: the KPI fully satisfies this criterion. As a result of this process, each KPI will receive a score from 0 (minimum score) to 10 (maximum score). The evaluation can be performed by expert teams who have extensive experience in the design evaluation/monitoring frameworks and oversee the implementation of the project. Then, the KPIs with the highest score will be included in the final KPI repository. A cut-off rule of a minimum score of ‘X’ points can be set for all KPIs to be considered for selection; this can change according to the needs of the evaluator. In case two KPIs serve the same purpose, the one with the highest score should be selected, whereas in the case of equal scores the KPI with the highest score in relevance should be selected. The following criteria should be further considered while selecting KPIs:

• Completeness: The set of KPIs should consider all different aspects of the project’s scope. In that respect, KPIs must be selected to cover all defined targets and different stakeholders’ perspectives;
• Non-redundancy: The set of KPIs should not measure the same aspect of a subtheme. Extra care should be given to not include indicators that assess the same parameter (double counting), even if the score was higher in comparison with other indicators;
• Independence: Small changes in the measurements of an indicator should not influence preferences assigned to other indicators in the evaluation.

Following this process, the number of KPIs to be included in the repository can be significantly reduced. In the end, it is highly recommended to iterate with key stakeholders in order to define the final KPI repository. Although the above-mentioned criteria are especially important for KPIs’ prioritization, it is also of major significance that these KPIs reflect the opinion and needs of the key stakeholders (e.g., end users and constriction companies). To ensure this, outcomes from the above-described steps should be shared and iteratively discussed with, e.g., construction managers and key technology providers who offer relevant solutions to be installed in the renovated buildings. In this way, KPIs that present a specific interest for the project’s stakeholders and were not included in the initial extensive pool of KPIs, can be integrated (or excluded) in the framework, leading to the finalized KPI repository. Once the KPI repository has been finalized, relevant KPI cards should be developed (Step 4). KPI cards include all necessary information for the understanding and estimation of KPIs included in the final KPI repository. KPI cards aim to provide key insights into what-how-when needs to be measured. Information that can be provided per card (indicative recommendations) is provided in Figure 3.

Figure 3. Example of a KPI card and its contents.

4. Case Study

The methodology described in Section 3 has been implemented in the Horizon EU project InCUBE for the determination of the final KPI repository upon which the project is going to be evaluated. InCUBE brings together 23 high-profile partners and two affiliated entities from seven EU countries, envisioning unleashing the EU renovation wave through cutting-edge, standardized, lean integrated processes based on four key pillars of innovation:

• Industrialization: off-site manufactured solutions including the use of robots (e.g., demolition, telescopic cranes, drilling, and waste sorting), so far applied only in industrial environments, offering novel services (e.g., anticollision, area boundary, waste tracking);
• Novel self-RES power producing and storage technologies, products and environmentally friendly materials (e.g., low GWP-refrigerant heat pumps);
• Digitalization: dynamic digital twins of both products and buildings, utilizing immersive capturing techniques (e.g., laser 3D scanners and drones), digitally merging innovative manufacturing processes with BIMs, and;
• New market entrants, organized under novel business models, to allow for increased levels of collaboration and productivity.

The InCUBE Suite integrates digital tools across all four pillars and enables the seamless coordination of different renovation phases while leveraging data streaming from multiple interoperable sources to accommodate tenants’ comfort and render buildings’ active energy nodes in the synergetic energy networks paradigm of the future. Solutions will be validated in three large-scale demo sites in three countries (Italy, Spain and the Netherlands), including a cultural heritage building. Key expected outcomes include, among others, reduction of waste streams and time needed on-site, renovation cost reduction, reduction of working time with hazardous activities, energy savings, GHG reductions, achieving a reduced energy performance gap between as-built and as-designed, and creation of RECs, all while accounting for social inclusion, upskilling, the and enhancement of women’s role in the construction industry.

4.1. Step 1: Develop an Initial Pool of KPIs Building upon InCUBE’s Expected Outcomes

InCUBE will address several layers of innovation activities: technologies, process, operational, and business model innovation, taking a systems approach to tackling building stock decarbonization. InCUBE, in line with Built4People [52], brings together the whole construction value chain to accelerate people-centric innovation in the built environment, driving the transition towards a sustainable society and economy. This will be achieved through the definition and implementation of three (3) key impact pathways (KIPs) addressing scientific, societal (inc. environmental), and economic/technological impact in full alignment with HORIZON EUROPE Legislation [53]. These KIPs present the links and interconnections among the project’s expected results and outcomes. InCUBE outcomes have been clearly defined in the Grant Agreement (GA) of the project and some indicative examples are presented in the following table (Table 2).

Table 2. Indicative examples of InCUBE’s expected outcomes.

Call Expected Outcome (As Imposed by the HORIZON Program)	InCUBE Outcomes (Specifically Incurred by InCUBE)
Significant improvement in productivity of construction and renovation processes for energy-efficient buildings, supporting an increase in scale in the renovation process and streamlining resource efficient nearly-zero-energy performance renovation: 30% waste reduction; improved quality of renovation; at least 30% and towards 50% reduction of on-site construction/renovation work time; and 25% cost reduction	Scientific
	Creation and use of high-quality new knowledge on issues relevant to work optimization, compliance checking, on-site automation, off-site manufacture
	Economic/Technological
	Reduction of waste streams and time needed on-site for construction/renovation. Reduction of construction/renovation costs
Enhanced quality of construction, backed up by post-occupancy evaluations, also supporting better integration of design and construction activities, streamlining commissioning of buildings, in particular concerning energy management but also considering cross-cutting issues, such as accessibility of buildings	Societal/Environmental
	Accessibility of buildings: Likert scale—4.5/5.0 (very high). Buildings are designed to be user-friendly for everyone including people who use wheelchairs, canes, and those with vision and hearing impairments
	Economic/Technological
	Increased automation during construction and renovation. Reduction of snagging and minor defects compared to conventional construction practices

InCUBE expected outcomes, have been translated into relevant KPIs, which are presented in ANNEX I. In total, 34 KPIs have been defined that fully cover InCUBE’s short/medium-term scope and are fully aligned with InCUBE’s expected outcomes. From these KPIs, one (1) addresses the scientific KIP, fourteen (14) address the societal (incl. environmental) KIP, and nineteen (19) address the economic/technological KIP. The majority of these KPIs (18/34) must be evaluated on a project level (considering the accumulated impact of all InCUBE activities), whereas the rest (15/34) are evaluated at the demo-site level and one (1) on a technology level.

4.2. Step 2: Expand the KPI Pool by Capitalizing on Information from Other Sources

The following process has been adopted per source to identify KPIs that can be utilized to update InCUBE’s KPI repository. Regarding building deep-renovation HORIZON frameworks, a search was conducted using the Cordis EU platform. A filtering procedure was applied to reduce the vast number of available projects. Only HORIZON projects starting from 2018 and beyond were included in the analysis to ensure that the information available is up to date. The following keywords were also applied in the search engine: “building”, “deep”, and “renovation”. Cordis returned 40 results—projects meeting the pre-mentioned criteria. These projects were quickly evaluated (e.g., by checking their abstracts and websites) to examine their relevance with InCUBE scope, and if relevant were catalogued in a list. For projects included in this list, a more detailed search was conducted to find if there are public deliverables including information regarding their evaluation framework, assessment methods utilized, and specific KPIs proposed.

In total, four (4) projects were identified, which are highly relevant to InCUBE; their KPIs are publicly available. The key information for these projects is presented in Table 3. The KPIs included in these deliverables were recorded and, finally, further assessed for their suitability to be included in the InCUBE KPI pool. KPIs that were already covered by the initial pool developed in Step 1 were not considered for inclusion, as well KPIs that dealt with very project-specific issues (addressing only one project unless this was also highly relevant for InCUBE). Extra emphasis was placed on avoiding double counts. This initial evaluation was performed by the members of InCUBE’s Scientific and Technical Management Team. In total, 116 KPIs were reviewed, of which 92 were unique (not considering KPIs used in more than one project). From those 92 KPIs, 12 KPIs were selected as of interest to be included in the InCUBE KPI pool, considering the above-mentioned factors. A first key observation is that many of the KPIs utilized from these four projects were already defined in Step 1.

Table 3. Indicative examples of InCUBE’s expected outcomes.

Project ACRONYM	Brief Description	Total Number of KPIs
RINNO	RINNO targets building deep renovation by addressing key technical and socio-economic inhibiting factors through the development of innovative technologies, processes, tools, and business models. The monitoring framework for the evaluation of RINNO solutions covers five domains: technical, economic, environmental, materials, and social aspects. The developed KPIs are based on the specific project requirements and are evaluated using the “SMART” and “RACER” criteria.	61
StepUP	StepUp aims to develop an innovative deep renovation methodology based on the utilization of building performance data and physics-based modelling. Key project outputs include plug-and-play technologies targeting energy, costs, indoor environmental quality, and user comfort that will lower renovation investment risks and increase its benefits for the end users. The project KPIs reflect the financial, energy, and quality aspects of renovation and provide the guidelines for a holistic renovation evaluation.	19
BIM-SPEED	BIM-SPEED focuses on the widespread adoption of BIM to reduce deep renovation times. The project’s objective is tackled through the provision of an affordable BIM cloud platform and corresponding tools, while interoperability among the BIM tools is also investigated. The use cases of the project are evaluated based on a set of KPIs capitalizing on previous H2020 projects, as well as the EU Level(s) evaluation framework, targeting mostly the aspects of energy, environment, comfort, and affordability.	14
BUILD UPON 2	BUILD UPON 2 provides a supportive impact framework to facilitate the deep renovation acceleration of the EU building stock. The framework focused on the revised Energy Performance of Buildings Directive to decarbonize buildings by 2050, and will allow the measuring of progress in strategy building. It will be tested in eight pilot cities, and the feedback will be incorporated to help authorities at all levels meet their EU energy efficiency obligations.	22

InCUBE aims to support compatibility with EU Strategic plans and initiatives as much as possible, especially: (i) Built4People (B4P), (ii) Level(S), and (iii) Smart Readiness Indicator (SRI). Following a similar approach to the HORIZON frameworks, valuable KPIs can be extracted from these sources. B4P, Level(S) and SRI (see Section 2) were comprehensively reviewed and KPIs were identified and analysed. The progress of the B4P partnership’s activities is proposed to be tracked utilizing a set of nine (9) B4P-specific KPIs. From these nine KPIs, three (3) KPIs (established innovation clusters, training capacity (in hours per year), and new skills creation) have been selected to be included in InCUBE’s KPI pool. The Level(S) framework is assessed by using 16 KPIs linked to specific macro-objectives. From these 16 KPIs, 11 KPIs have been selected to be included in InCUBE’s KPI pool. Regarding SRI, a KPI that requires the complete quantification and evaluation of the total SRI score has already been included in the InCUBE initial pool (addressing Call EO#3); thus, no further additions are required.

InCUBE also draws KPIs from the publications in scientific journals presented in Section 2. The analysis of the studies showed that there is no universal agreement on the aspects that need to be considered when evaluating the sustainability performance of a renovation project, either at the building or cluster of buildings scale (neighbourhood/district). Most commonly, the environmental, economic, and social dimensions of sustainability are considered (triple-bottom approach) when assessing the performance of building/district renovation projects, whilst several recent studies suggest additional aspects have emerged for which relevant KPIs have been defined. The different aspects presented in Jafari and Valentin (2018); Toufeili et al. (2019); Terés-Zubiaga et al. (2020); McGinley et al. (2022); and Ascione et al. (2022) are already being addressed by KPIs included in InCUBE pool, or the recommended level of detail is not essential or irrelevant for InCUBE. This is the first sign that the InCUBE KPI pool is already well-populated and includes most of the key aspects indicated by the literature. Three new KPIs were extracted from Urbinati et al. (2022) and Kylili et al. (2016) (six in total).

Standards relevant to InCUBE objectives and activities, provided by the organizations mentioned in Section 2, have been considered as well. Six (6) standards published by ISO have been identified as relevant to the activities of InCUBE. Out of the identified as relevant standards, two (2) are the ones that offer potentially valuable information for the assessment of InCUBE activities: ISO 21931-1:2022 [33] and ISO 20887:2020 [35], which happen to be the most recent ones and provide information on how to assess construction performance on several different levels and guidance towards achieving adaptability and the process of disassembly, respectively. Open access to the presented standards is not available; thus, potentially useful information for the development of the InCUBE KPI pool could not be examined in detail. Regarding CEN-CENELEC standards, standard CEN/TR 17005:2016 [29] in particular includes suggestions regarding the assessment of the environmental performance of a building, including relevant indicators, and could be of value to the evaluation of the environmental performance of the InCUBE pilots. Additionally, standards EN 16627:2015 [30], EN 16309:2014+A1:2014 [31], and EN 15978:2011 [32] provide calculation methodologies for the evaluation of the economic, social, and environmental performance of the building, respectively, that could offer guidance for the respective InCUBE framework. However, the standard documents are not publicly available and, therefore, as in the case of ISO standards, were not taken into consideration in the expansion of the InCUBE KPI pool, since InCUBE opted to build upon open-access data only to support replication. ETSI indicators that can be utilized for or adapted to a building/project level were selected from an indicator pool including, among others, KPIs deriving from the relevant CITYkeys [48] deliverable. The indicators are divided into various categories and some of them can be adapted to monitor sustainability performance on a building and/or project level, as well as city-wide. Two (2) of these KPIs dealing with data privacy and accessibility of data have been selected to be included in InCUBE’s pool.

The implementation of Steps 1 and 2 resulted in an extended KPI pool including 68 KPIs, summarized in ANNEX I. Almost half (34) of these KPIs were extracted to fully facilitate the monitoring of all InCUBE outcomes and eventually validate the success of InCUBE solutions. The rest of the KPIs were extracted from highly relevant Horizon frameworks (12), B4P partnership (3), Level(S) (11), publications in scientific journals (6), and ETSI standards [39] (2). From these KPIs, one (1) addresses the scientific KIP, thirty-four (34) address the societal (incl. environmental) KIP, and thirty-three (33) address the economic/technological KIP. The majority of these KPIs (41/68) have to be evaluated on a demo-site level, whereas the rest (25/68) are evaluated at the project level (considering the accumulated impact of all InCUBE activities) and two (2) on a technology level (per innovative solution as defined in InCUBE). This pool of KPIs can serve as a great starting point for other deep-renovation and building smartification-oriented projects looking to identify specific KPIs to apply.

4.3. Step 3: Refine the KPI Pool Based on Pre-Defined Criteria and Tailored Feedback

The implementation of Step 3 resulted in a shortlisted KPI pool which eventually defined the final InCUBE KPI repository. The KPI pool presented in Section 4.1 was evaluated by using the criteria (relevance, availability, measurability, reliability, and familiarity) mentioned and the process described in Section 3. The evaluation was performed by InCUBE’s Scientific and Technical Management Team (STM). Scores indicate a commonly agreed final score of all members, after conducting an internal meeting where all STM members examined the KPIs. Resulting from this process, the number of KPIs included in the InCUBE KPI pool was reduced to 27 from the initial 68. The scores are available in Table A1 in Appendix A.

A quick statistical analysis of the scores concerning the initial 68 KPIs was conducted to provide insights into the selection process and the quality of the selected KPIs, overall, based on the evaluation criteria. The average final score among all initial KPIs was found to be 6.29, indicating that most of the initially selected KPIs were fairly suitable for the project results’ evaluation, owing to the relevance of InCUBE to the examined frameworks and projects that provided the additional KPIs included in the pool. More specifically, averages for each separate criterion were calculated to be as follows: 1.72 for relevance, 1.26 for availability, 1.15 for measurability, 1.13 for reliability, and 1.03 for familiarity. The highest average score is assigned to relevance, indicating that the EO-oriented process of selection was efficient in this respect, while the lowest score, familiarity, suggests that the selected KPIs lack clarity and ease of understanding from the perspective of a non-expert. KPI cards will help address this shortcoming. Among the three (3) key impact pathways, the scientific pathway scored the highest (7 points) but only included one (1) KPI. The societal/environmental pathway scored an average of 5.91 points, while the economic/technological pathway’s average score was 6.67. The difference between societal/environmental KIP and economic/technological KIP is that social issues are harder to quantify measurably.

The scored version of the KPI pool was communicated to pilot managers and key technology providers of InCUBE for their review. The following feedback was received:

• KPI #3 “Accessibility of the building” was indicated as of high importance by social inclusion experts, and is therefore included in the final list. This was also considered a key parameter that needs to be monitored in the pilots.
• KPI #7 “Women empowerment in the construction industry” was suggested to be included as it relates closely to the project’s expected outcomes (EO#6). The initial scores considering measurability and familiarity were increased by one point after suggestions by InCUBE’s social experts.
• KPI #14 “Medium-term GHG emissions (during operation)” was suggested to be integrated with KPI #57 “Yearly Life Cycle GWP savings”, since this is overlapping.
• KPI #20 “Usability of the building while replacement tasks are carried out” was considered to be critical for the ES and NL pilots and is therefore included in the final KPI list.
• KPI #38 “Total renovation costs” was removed from the final list, as it was considered to be a part of KPI#40 “Payback time of the renovation project as a whole”, and monitoring it separately was deemed redundant.
• KPI #55 “Yearly primary energy savings” was considered important by technical partners reviewing the initial list and it was suggested that the familiarity score could be increased it was therefore added to the final list.
• It was suggested that the familiarity score of KPI #56 “Life cycle costs” was too low since maintenance costs as part of the life cycle costs are a commonly understood concept.
• KPI #61 “Change in value of the property” was considered important, especially for the ES pilot, and was also included in the final KPI list.
• Finally, considering that many of the project’s interventions aim at the reduction of the pilots’ heating demand, the inclusion of a KPI measuring the “Heating demand reduction” linked to the renovation activities, was suggested and added as a new KPI.

This feedback was integrated into the scored KPI pool, leading to the final InCUBE KPI repository, which includes 31 KPIs (highlighted in Table A1 in Appendix A).

4.4. Step 4: Develop KPI Cards for the Final KPI Repository

Finally, KPI cards per KPI included in the final InCUBE repository were developed. Below in Figure 4 an example is provided.

Figure 4. Example of a filled-in KPI card.

The data required to calculate the various KPIs may come from a variety of sources; some are measured from sensors (e.g., electricity production/demand, temperature), and other data from surveys. For data from surveys (tenants, construction workers), a standard form with a list of predetermined questions and potential answers, using Likert scale, can be utilized to calculate the KPIs. Following data collection, specific criteria are used to assess data quality. Pre-processing and imputation techniques, for example, can be used on automated data because there may be missing values, outliers, and discontinuities due to inconsistent electrical power operation or incorrect measurements from smart sensors.

5. Conclusions

Sustainability is increasingly important in the decision-making process for deep renovation projects throughout their lifecycles, so it is necessary to identify pertinent metrics to track the sustainability performance of these projects. This study suggests a method that can be applied for making the monitoring process consistent and transparent. The four-step methodology proposed improves the quality and appropriateness of the chosen KPIs while ensuring that significant issues important to the sustainability of a building are reflected during the evaluation. This is achieved by starting with tailored KPIs (Steps 1 and 2) and then scoring them in a very specific way with universal criteria (Step 3). Step 4 includes the creation of the KPI cards, which provide all the necessary information for understanding and estimating the KPIs included in the final repository. This methodology prevents the monitoring process from becoming unmanageably complex by selecting only the most pertinent and appropriate KPIs in the end. The InCUBE project’s recommended sources and preidentified KPIs are available to interested stakeholders (such as other academics, evaluators, and project managers) to facilitate and expedite the monitoring of their projects. Future research can concentrate on how to combine various indicators (e.g., extraction of indexes and subindexes), while the application of the methodology in different contexts (e.g., building typologies and climate zones) could help in examining the differences in selected KPIs and underlying evaluation needs, taking into consideration the differences in construction processes and regulations between different EU countries.