Weighting Key Performance Indicators of Smart Local Energy Systems: A Discrete Choice Experiment

Abstract

The development of Smart Local Energy Systems (SLES) in the UK is part of the energy transition tackling the energy trilemma and contributing to achieving the Sustainable Development Goals (SDGs). Project developers and other stakeholders need to independently assess the performance of these systems: how well they meet their aims to successfully deliver multiple benefits and objectives. This article describes a step undertaken by the EnergyREV Research Consortium in developing a standardised Multi-Criteria Assessment (MCA) tool—specifically a discrete choice experiment (DCE) to determine the weighting of key performance indicators (KPIs). The MCA tool will use a technology-agnostic framework to assess SLES projects, track system performance and monitor benefit realisation. In order to understand the perceived relative importance of KPIs across different stakeholders, seven DCEs were conducted via online surveys (using 1000minds software). The main survey (with 234 responses) revealed that Environment was considered the most important criterion, with a mean weight of 21.6%. This was followed by People and Living (18.9%), Technical Performance (17.8%) and Data Management (14.7%), with Business and Economics and Governance ranked the least important (13.9% and 13.1%, respectively). These results are applied as weightings to calculate overall scores in the EnergyREV MCA-SLES tool.

1. Introduction

Smart Local Energy Systems (SLES) are being developed to connect various energy vectors (e.g., transport, heat, and power) through flexible energy supply, demand and storage options by exploiting digital technology and the Internet of Energy [1,2]. The deployment and development of SLES has the potential to resolve the energy trilemma (producing cleaner energy at an affordable price with acceptable energy security) [1,2]. Furthermore, SLES can provide many co-benefits that progress towards 11 out of the 17 United Nations (UN) Sustainable Development Goals (SDGs) [2,3]. SLES can provide cleaner, affordable energy, resilient infrastructure, job creation and improved living conditions, which correspond to SDG7 Affordable and Clean Energy, SDG9 Industry, Innovation and Infrastructure, SDG8 Decent Work and Economic Growth and SDG11 Sustainable Cities and Communities [4]. Enabling the delivery of these benefits is a key driver for ongoing financial investment in SLES. To ensure that this potential of SLES is realised, however, investors and other stakeholders need to be able to measure the success and performance of an SLES project to understand what works, for whom and in what context.

This article describes a set of discrete choice experiments (DCE) used in the development of a Multi-Criteria Assessment (MCA) tool to specifically focus on SLES, which is being developed by members of the Innovate-UK-funded EnergyREV project. MCA methods have been applied to carry out a wide range of analyses on complex energy planning and strategy issues. These have provided information to enable the energy transition thorough improvements in decision making, policy design, development strategies and frameworks [5,6,7,8,9,10,11]. The MCA carried out by Heo et al. [5] resulted in an increase in renewable generation capacity in Korea from 2.8% in 2007 to 11% by 2030, while another MCA has identified which technologies should be prioritised in future energy policies and strategies for Lithuania [6]. In Moldova, which is considered to be an energy-deficient country due to its limited energy resources, MCA has also been applied to inform the direction of future energy system development [7]. MCA has also been used to understand potential local societal acceptance of energy technologies and systems (of value to policy making) by ranking different energy system scenarios according to local stakeholder preferences in the Faroe Islands [8].

Despite these existing MCAs of energy systems, there is currently no standardised approach to assess SLES performance, and most existing tools are not completely suitable for the purpose; they may be focused on techno-economic metrics, or be complex and difficult to use for this application (this is described in greater detail in [12]). The authors are, therefore, developing a simplified, technology-agnostic MCA tool to examine SLES projects that will track both the system performance and the benefits that may be realised. This independent, standardised assessment tool will support SLES project developers in bench-marking progress against project aspirations, aid in gathering evidence to build investors’ confidence and, over time, be used as a route map and checklist for SLES replication and expansion. The tool is also expected to assist policymakers in identifying areas where policy change is required in order to enable progress.

The first step in developing this MCA tool for SLES was to identify the main criteria for success (or failure) of a project and the corresponding metrics to measure them. The details of this process (described in [12,13] and summarised in Section 1.2 of this article) drew on the existing literature and evaluation tools, alongside public stakeholder workshops. The process resulted in a comprehensive list of key performance indicators, which were grouped into six key themes: “technical performance, data management, governance, people & living, business & economics, and environment” [3]. The key performance indicators identified during this process included both primary benefits (or core outcomes) and support solutions (e.g., data management and governance) critical for delivering SLES objectives.

The next step in combining these key performance indicators into an MCA tool is to characterise their relative importance via a set of weights, and this is the focus of this article. These weights are identified by collecting public stakeholder views using a discrete choice experiment (DCE) refined by means of semi-structured interviews with subject experts. The DCE method is commonly used in research related to understanding consumer choice [14,15,16]. In recent years, DCE has been used more frequently in research concerning the energy transition to capture stakeholder preferences related to key components of this transition. These include different energy policies and energy technologies [17,18], the identification and selection of assessment indicators for energy systems [14,16], energy policies related to local energy communities and social acceptance [19] and green infrastructure [20]. The study by Azarova et al. [19], for example, applied a choice experiment approach to analyse the attitude of local communities towards different configurations of renewable energy technologies within the local energy system in four European countries. This found that energy system developments focusing on gas power plants and increased power transmission infrastructure had low social acceptance, while those with increased solar implementation and power-to-gas infrastructure development had high social acceptance.

There is limited application of the DCE method for identifying and determining the relative importance of energy assessment indicators; only two similar pieces of research have been identified to date. Both Naegler et al. [14] and Hottenroth et al. [16] applied the DCE method to capture and analyse stakeholder preferences towards sets of indicators with the purpose of determining which to include (or exclude) for different assessment scenarios and energy transition pathways, and the corresponding weighting.

This article aims to contribute to the ongoing discourse on applying DCE to determine the weights of criteria for MCA of energy systems from stakeholder preferences. The position of the DCE within the process of the MCA-SLES tool development is described alongside details of the previously defined assessment criteria and the resulting criteria weights. The DCE was conducted via an online survey to understand the dynamics of multiple components of SLES among various stakeholders. Participants were asked “what kind of energy system do you prefer?”; this simple question facilitated ranking the relative importance of various metrics within the six themes (Technical Performance, Data Management, Governance, People and Living, Business and Economics and Environment).

The remainder of the article is structured as follows: in the rest of Section 1, background information is provided on possible approaches for measuring the relative importance or weighting of key performance indicators for multiple objectives of a system, and the previous work conducted in specifying the criteria used to assess the success or failure of SLES is summarised; in Section 2, the design of the DCE survey to elicit the energy preferences of various stakeholders to determine the relative weightings is presented, with the results discussed in Section 3; and, finally, Section 4 comprises concluding remarks and recommendations for further work.

1.1. Multi-Criteria Decision Making

The MCA tool under development is based on MCDM methodology. Multi-criteria decision making (MCDM)—also known as multi-criteria decision analysis (MCDA) and increasingly performed using specialised software—is a methodology to support decision making when there are multiple criteria or objectives to consider in ranking or choosing between alternatives. Weighted-sum models are widely used for evaluating and aggregating these trade-offs between criteria. Other methods, that are not based on aggregative weight-based functions, include the “outranking” methods group (e.g., VIKOR, ELECTRE and PROMETHEE) and fuzzy methods, which are considered relatively more complex in comparison with the weighted-sum model [21,22].

The MCA-SLES tool employs a weighted-sum model. Consequently, understanding the relative importance of the assessment criteria for energy system transition, technology development and development in relation to energy policies and strategies is critical [6,7,8,9,11]. Capturing relevant stakeholder (i.e., energy providers, system and project developers and local and national government agencies) perspectives to determine this relative importance improves the reliability and relevancy of the MCA application, particularly when it comes to assessing location or sector-specific projects, benchmarking progress, highlighting the potential benefits and delivering information useful for gaining financial, political and public support [6,8,9,11,23].

Although there are several types of MCDM selection methods, no particular method has a distinct advantage or disadvantage over the other [24,25]. The ease of use and understanding, confidence in the results and reliability (consistency) are primary concerns that normally dictate the selected MCDM method [26].

No matter the approach, there are several generic steps involved in the MCDM process [21,27].The process, described in detail by Hansen and Devlin [22], is summarised here:

• Structure the decision problem and identify output;
• Specify the relevant criteria or indicators;
• Measure the performance of alternatives;
• Score the alternatives according to their impact on the criteria;
• Weight the individual criteria;
• Rank the alternatives based on scores and weights;
• Apply the outputs to support decision making.

The research presented in this article is focused on the development of a standardised MCA or MCDM methodology, which will be tested and refined by application to real SLES in future work. In particular, this article describes how the scoring systems and criteria weights were identified and defined for application in steps 4 and 5 of the MCDM process. These steps are intrinsically linked and, in essence, determine the validity and reliability of the MCDM outputs. The definition of the problem and associated criteria for steps 1 and 2 of this research are described briefly in Section 1.2 and in greater detail in Francis et al. [12]. Step 3 is a practical step for which standardised methodologies will be further developed through application of the MCDM process to a real SLES in future work.

1.2. Criteria for a Smart Local Energy System

Smart Local Energy Systems can be considered a system of networked systems and are socio-technical by nature [28]. A complete assessment of the performance and benefits realised from SLES projects must, therefore, examine the socio-technical environment combined with an integrated assessment of the multiple factors driving the low-carbon transition.

As mentioned in Section 1, one of the preliminary steps in designing the MCA tool for SLES was to identify the main criteria for success (or failure) of a project and the corresponding metrics to measure them. This was achieved by a combination of exploring existing multi-criteria assessment protocols for related applications and gathering data via a series of stakeholder consultations. Even though there were overlapping evaluation methodologies, four main analytical themes were identified in the literature (summarised in Figure 1):

• Maturity or Readiness Level—Considering the readiness or maturity of a product and/or service, including: Technology Readiness Level—a de facto standard assessment tool used in aerospace, defence and technology [29]; Technology Performance Level—used to assess wave energy converters; or the Energy Transition Index—used to assess and compare electricity flexibility markets and determine their preparedness for energy transformation [30].
• Planning and Forecasting—Incorporating multiple criteria, such as the technical, economic, environmental and social influences of a product and/or service for planning or forecasting. For example, integrated assessment modelling—for evaluating sustainable energy systems MCDA, optimisation models and software tools) [31]—or the techno-ecological synergy (TES) framework—implemented to improve the sustainability of solar energy across four environments: land, food, water and built-up systems [32].
• Sustainability Transition—Considering the sustainability transition of products, services, processes, people and overall networked systems in their environments across multiple objectives. These include socio-technical transition frameworks, namely a multi-level perspective—which considers the alignment of the incumbent regime, radical “niche innovations” and the “socio-technical landscape” [33]—and strategic niche management—which facilitates the creation of protected spaces for experimentation on: the co-evolution of technology, user practices and regulatory structures [34].
• Other—Miscellaneous tools and indicators that have been used to measure the smartness and/or sustainability of homes, the electricity grid [35], cities [36,37,38,39] and integrated community energy systems (ICES) [40], as well as procedures involving sustainable accounting of six capitals—financial, manufactured, intellectual, social and relationship, human and natural—for assessing long-term viability of an organisation business model [41] and could be applied to the assessment of SLES.

From the analysis of the literature, augmented by data collected from stakeholders through two facilitated workshops, a number of common themes and indicators emerged which could be adapted to assess the performance of SLES. A total of 50 relevant performance indicators were identified and were clustered into 10 key themes (Appendix A). These themes and indicators have previously been applied in the assessment of sustainable energy, smart energy, smart grids, smart cities and renewable energy products, services or systems.

These themes and indicators were proposed as the basic structure of the taxonomy for the MCA (i.e., themes, sub-themes, success criteria and metrics). They were reviewed by participants in a third stakeholder workshop, and there was a consensus that they could be merged and simplified into six themes; for example, data security and data connectivity were merged into data management. These six themes, used to classify the performance, multiple benefits and consequences of SLES, are shown in Figure 2 and defined as follows:

• Data Management—Data gathering and security, provision of ICT and data infrastructure, including issues such as ICT accessibility and penetration
• Technical Performance—Technical performance, including indicators such as resilience, efficiency and innovation. All vectors: heat, power and transport.
• Business and Economics—Financial and economic performance, such as benefit-to-cost ratio, rate of return, financing, job creation and socio-economic impacts.
• Governance—The political and regulatory environment, including alignment with existing regulations and their interface with policy.
• People and Living—The impact on end users (education, ICT skills, engagement or acceptance) and their associated benefits on communities and social interactions (equity, housing conditions, culture or behaviour).
• Environment—The environmental performance, namely the impacts on climate change, human health, resource availability and use of waste energy.

The list of key performance indicators within these themes identify both primary benefits (core outcomes) and support solutions critical to SLES delivery. The provision of functional support solutions such as Data Security and Governance should be monitored to identify whether key boundary conditions are met and ensure that unintended negative consequences or impacts are avoided.

The alignment of the key themes and indicators with the United Nations SDGs was also identified, so that the wider co-benefits can be tracked. Additional details of this work are outlined in [12].

The next section outlines the methodology used to score and weight the themes and indicators.

2. Methodology

A discrete choice experiment (DCE), conducted via online surveys, was designed to reveal the preferences of the various stakeholders in SLES with respect to the relative importance of the criteria (i.e., themes or indicators, as defined in Section 1.2). The resulting weights on the criteria (sometimes called “part-worth utilities” in the DCE literature [42]) can be used as a practical rating and scoring instrument for the MCA-SLES framework. An advantage of the DCE method used in this study is that it generates a set of weights for each individual participant, which enables cluster analysis—wherein any “clusters” (or segments) of participants with similar patterns of weights can be identified [42,43].

2.1. The PAPRIKA Method

The DCE was undertaken using 1000minds software (www.1000minds.com). This applies the PAPRIKA method—Potentially All Pairwise RanKings of all possible Alternatives [44]. The PAPRIKA method involves capturing preferences by asking stakeholders to repeatedly choose between two hypothetical alternatives defined on two criteria at a time (i.e., “partial profiles”). From these choices (or pairwise rankings), scores and weights are indirectly derived using quantitative methods [42,43]. In contrast, other types of DCE—also known as conjoint analysis—are regression-based [45].

In this analysis, the PAPRIKA pairwise-ranking questions involved a choice between two hypothetical SLESs, defined on two criteria at a time and involving a trade-off between them. These criteria describe a particular characteristic of the SLES, usually related to a success attribute or metric (indicator), e.g., quality of performance. For each criterion the performance is described by multiple levels on a defined scale; for example, the Governance of the SLES would be a criterion with possible performance levels ranging from poor to excellent. Details of the pairwise-ranking questions are given in Section 2.3, with comprehensive lists of the criteria and levels in the Supplementary Information.

The descriptions of the levels were carefully chosen to ensure that the surveys were not asking leading questions but were expressed in an open-ended way, such that participants provided answers based on their personal judgement. The 1000minds software includes features to check for the consistency and reliability of participants’ answers, enabling participants who answered the questions inconsistently or too quickly (so that they were deemed to be unreliable) to be identified and excluded.

2.2. Overview of Surveys

In this study, a total of seven surveys were prepared: a main survey (mandatory for all participants) to ascertain the relative weightings of the six themes (i.e., Technical Performance, Data Management, Governance, People and Living, Business and Economics and Environment) and a further six optional surveys to independently examine the relative weighting of the key performance indicators (KPIs) within these six themes in greater detail. Participants were invited to complete one or more of these latter theme-specific surveys according to their area of expertise. Including the mandatory main survey, participants were typically expected to complete two surveys out of the seven. It was anticipated that each survey would take around 8–10 min to complete; however, in some cases, it could take longer depending on the options selected by the participant and the extent of their deliberation and engagement.

Participants were asked to answer the questions with reference to their main role in the energy sector. They were also asked to declare what this stakeholder role was, selecting from: Government, Non-Governmental Organisation (NGO) or Non-Profit Organisation (NPO), Regulators, Community Energy, Large End User, Small End User, Product Manufacturer and Retailer, Finance Sector (banks and funding schemes), Network Operator and Advisors (cooperatives, consumer support), Research Organisation or University, Industry (generation, transmission, distribution and retail), Local Authority, Consultant and Other.

The surveys were opened for a month from 26 January to 8 March 2021 and distributed to approximately 1500 individuals via emails and mailshots to various member list groups in academia and industry. The surveys were also promoted through social media platforms such as Twitter, LinkedIn and Facebook. A small incentive of ten £50 Amazon vouchers randomly given away in a prize draw was offered to encourage individuals to participate in the surveys. The next section describes the main and additional surveys.

2.3. Main Survey

The main survey was designed to determine the relative importance of six KPI thematic areas that will be used to assess the performance and benefits of SLES. The participants were presented with a pair of hypothetical SLES alternatives that were the same, except for a trade-off in the different levels of two criteria, and asked to indicate which SLES they preferred (see Figure 3). The main survey had six criteria, which correspond to the six KPI themes:

• Technical Performance;
• Data Management;
• Governance;
• People and Living;
• Business and Economics;
• Environment.

Performance on each criterion is measured using five levels:

• Poor;
• Fair;
• Good;
• Very good;
• Excellent.

The 1000minds software presents the survey as a series of flashcards contrasting pairs of hypothetical SLESs in a randomised order, and participants must indicate which one they prefer. For a survey such as this, with six criteria, each with five levels, participants would typically be presented with 27 choices. A large majority of the participants in this study (90%) completed this survey in 4–12 min, with the remainder taking 20–40+ min.

PAPRIKA exhibits “path dependency” because of its adaptive nature: the method chooses questions for the participant based on all preceding answers [42]. Thus, the PAPRIKA method is a type of adaptive DCE (or adaptive conjoint analysis) [44,45]. One example of this is in the application of the logical property of "transitivity" to minimise the number of questions each participant is asked [42,43]. Each time a person ranks a pair of SLES, the PAPRIKA method immediately identifies and eliminates all other pairs of hypothetical SLES for which the ranking can now be inferred; for example, if a person ranks System 1 ahead of System 2, and also 2 ahead of System 3, then, logically, transitivity shows that 1 must be ranked ahead of 3. This third pair of systems is thus eliminated from the questioning process. Through this process, a relatively small number of questions (e.g., 27) can be asked to rank all hypothetical systems differentiated on two criteria at a time, either explicitly or implicitly (through transitivity).

The participant’s choices from the pairwise-ranking process are used to calculate their assessment of the relative importance weights (or part-worth utilities) of the criteria. This is achieved through mathematical methods based on linear programming, as described in Hansen and Ombler [44]. The tool also uses interpolation between levels to estimate weights, thereby further reducing the number of pairwise comparisons required.

To test the consistency (reliability) of the answers provided by the participants, two questions were repeated at the end of the DCE to check their overall "quality". In addition, participants who answered any question more quickly than 2 s were excluded (because they were deemed to be unreliable).

2.4. Thematic Surveys

A survey for each thematic area was also presented using the 1000minds software, applying the same approach described in Section 2.3 but concentrating on detailed indicators, or metrics. Participants were asked to complete one or more of these additional surveys. As before, participants were presented with a series of simple pairwise comparisons showing hypothetical alternatives of SLES that were differentiated by a trade-off in their criteria.

The criteria for these thematic surveys mostly corresponded to the detailed key performance indicators (success criteria, sub-theme or metrics) developed from an in-depth literature review [12] and data collected at the London workshop held in February 2020 [3]; described in Section 1.1. Each survey had six criteria (except for the People and Living survey, which had seven) and five levels.

Participants were asked to think about the performance of the SLES in respect of a specific KPI theme, for instance the governance and organisation of an SLES. They were then encouraged to select the three or four criteria within this theme that they considered to be the most important, in order to focus their time on ranking the indicators they considered the most important. This choice was not restricted, so it was possible for the participants to select as many as they desired; however, this would make the survey longer to complete.

Table 1. Theme-specific criteria (indicators, success criteria or metrics) for the six optional thematic surveys together with the typical number of pairwise comparisons presented to the respondent.

KPI	Criteria							Pairwise
Theme	1	2	3	4	5	6	7	Comparisons
Governance	Governance Strategy	Integrated Management & Digital Planning	Accountability & Decision Making	Transparency & Consumer Redress	Knowledge Exchange & Experience	Standards & Regulation		20
Environment	Greenhouse Gas Emissions	Biodiversity	Human Health	Resilience to Environment	Noise Levels	Other Ecosystem Impacts		20
Data Management	Digital Technology Enablers	ICT Infrastructure	Visibility	Privacy	Grid & Capacity Management	Investment Decisions		18
People & Living	Community Engagement	Fuel Poverty	Cost of Energy	Thermal Comfort	Access to Services	Carbon Reduction	Job Opportunities	17
Business & Economics	Market Design	Attractive to Investors	Competitive Energy Pricing	Promoting Growth	Revenue from Decarbonisation	Techno- Economic Metrics		34
Technical Performance	Robustness	Reproducibility	System Performance	Maturity	Energy & Infrastructure	Local Renewable Generation		15

shows the available criteria for each thematic survey, and the corresponding sub-criteria and levels presented to participants are given in the Supplementary Material.

The final number of pairwise-ranking questions in the DCE depended on the total number of criteria selected by the participant. Details of the average number of questions asked for each themed survey are also shown in Table 1.

In the resulting pairwise comparisons, most of the criteria used a five-point scale of poor, fair, good, very good and excellent; however, this scale was not found to be appropriate for all criteria presented. The level descriptions were carefully selected to match the context of the questions asked, avoid leading questions and to ensure that the participants’ responses were based on their personal judgement. This resulted in some specific alternative scales being used, as follows:

• Greenhouse Gas Emissions or Fuel Poverty:

  –

  Increased;

  –

  Remains the same;

  –

  Decreased;

  –

  Significantly decreased;

  –

  Eliminated (for Greenhouse Gas Emissions, this was termed “Achieves net zero (eliminated)”).

• Revenue from Decarbonisation Activities:

  –

  None;

  –

  £;

  –

  ££;

  –

  £££;

  –

  ££££.

• Local Renewable Energy Generation:

  –

  None;

  –

  A little;

  –

  Moderate;

  –

  Quite a lot;

  –

  Extensive.

• Competitive Energy Pricing (note the four-point scale):

  –

  More expensive energy;

  –

  Parity with today’s prices;

  –

  Slightly cheaper energy;

  –

  Significantly cheaper energy.

For a full list of the sub-criteria and scales used in all of the surveys, please see the Supplementary Material.

3. Results and Discussion

The DCE surveys were published and open for response from 26 January to 8 March 2021. All respondents were asked to complete the Main Survey plus at least one Thematic Survey.

Of the 387 people who responded to the survey request, 119 started to answer the main survey but did not complete it; 34 were excluded because they answered too quickly (less than 2 s per question); and the remaining 234 responses (60%) were used in the DCE analysis. Of these 234 respondents, approximately 47% were from research organisations and 16% represented small end users (e.g., householders and small business) and NGOs or NPOs. The energy industry and local authorities were each represented by 6–7% of the participants, as summarised in

Table 2. Different types of stakeholders who completed the survey.

Main Involvement in the Sector	Quantity	Percentage
Research Organisation or University	111	47.4
Small End User	37	15.8
Non-Governmental Organisation (NGO) or Non-Profit Organisation (NPO)	16	6.8
Local Authority	15	6.4
Energy Industry	14	6.0
Consultant	12	5.1
Community Energy	9	3.8
Other	9	3.8
Product Manufacturer and Retailer	5	2.1
Government	2	0.9
Finance Sector	1	0.4
Large End User	1	0.4
Network Operators and Advisors	1	0.4
Regulators	1	0.4

. Respondents were also asked about their relationship to the PFER programme and EnergyREV: 16% were affiliated to PFER demonstration and design projects, 29% were involved with other community energy projects and 15% were members of the EnergyREV Research Consortium.

3.1. Main Survey

The purpose of the main DCE survey was to identify the relative importance of the six KPI themes for SLES, and the resulting mean weights are summarised in Figure 4 alongside a sample of 10 results from individual participants. It can be seen that the individual results varied significantly in terms of the relative importance of different themes, but on average, Environment was considered the most important criterion, with a mean weight of 21.6%. This was followed by People and Living at 18.9%, Technical Performance at 17.8%, Data Management at 14.7%, Business and Economics at 13.9% and Governance at 13.1%. The standard deviation for each criterion ranged from 5% to 7%.

Overall, this DCE found that the benefits for people, their living conditions and environment were considered to be far more important by the stakeholders surveyed than the business and economic value. With regards to the environmental impacts, these findings compare well with the conclusions drawn by the MCA of renewable energy implementation in Lithuania [6], and the DCE carried out by Hottenroth et al. [16], which both concluded that environmental factors were key. They disagree, however, with Heo et al. [5], who found environmental factors to be one of the least important sets of criteria for the South Korean energy transition policy and development. With regards to the social factors (People and Living), the findings of this DCE contrast particularly with the MCA carried out by Štreimikienė et al. [6], where socio-ethics were found to be the least important criteria group. It is also surprising that business and economic factors were ranked so low by this DCE, as these were found to be of high importance in Heo et al. [5], Štreimikienė et al. [6], Hottenroth et al. [16]. Additional work is required to understand the importance of economics and market criteria in ensuring the economic feasibility of proposed energy system developments.

The variation in importance between the different KPI themes was found to be relatively small. This emphasises that it is vital for an assessment tool for energy system development to be holistic and encompass multiple factors across major themes in our modern society. In order to ensure a comprehensive and robust assessment, this should include people, their living conditions, the local-to-global market economy, technological and digital development and usability, governmental policies and strategy and the environmental impacts on local ecosystems.

3.2. Thematic Surveys

In addition to the main survey, participants were asked to complete at least one other DCE survey that focused on a specific KPI theme, in order to identify the relative importance and weights of each indicator within that theme. As these surveys were optional, fewer participants completed each of them than the main survey. The total number of completions (that were not excluded due to taking less than 2 s to answer a pairwise ranking question) is given alongside the results in

Table 3. Theme-specific KPI weightings calculated from the DCE thematic surveys.

KPI	Criteria Ranking and Weights							Included
Theme	1	2	3	4	5	6	7	Participants
Governance	Governance Strategy (23.3%)	Accountability & Decision Making (19.7%)	Standards & Regulation (16%)	Integrated Management & Digital Planning (15.2%)	Knowledge Exchange & Experience (13.4%)	Transparency & Consumer Redress (12.4%)		30
Environment	Greenhouse Gas Emissions (32.1%)	Other Ecosystem Impacts (20.3%)	Biodiversity (20.2%)	Human Health (17.1%)	Resilience to Environment (8.8%)	Noise Levels (1.5%)		56
Data Management	Grid & Capacity Management (20.6%)	Digital Technology Enablers (19.5%)	Investment Decisions (19.1%)	ICT Infrastructure (18.9%)	Visibility (13.2%)	Privacy (8.8%)		16
People & Living	Fuel Poverty (19.4%)	Carbon Reduction (16.5%)	Cost of Energy (15.1%)	Thermal Comfort (14.2%)	Community Engagement (12.6%)	Access to Services (11.7%)	Job Opportunities (10.5%)	51
Business & Economics	Market Design (22.3%)	Promoting Growth (21.4%)	Techno- Economic Metrics (15.5%)	Competitive Energy Pricing (14.8%)	Attractive to Investors (13%)	Revenue from Decarbonisation (13%)		31
Technical Performance	Robustness (26.6%)	Energy & Infrastructure (18.6%)	Local Renewable Generation (18.5%)	Reproducibility (13.0%)	System Performance (12.2%)	Maturity (11.1%)		44

. The full breakdown of participant completions and exclusions is given in the Supplementary Material.

It can be seen that the surveys on Environment and People and Living had the most responses, while Data Management had the fewest. It is also important to note that, as described in Section 2.4, at the beginning of each thematic DCE survey the participants were asked to select three or four criteria that they thought were most important (although they were were free to choose more). This was a means of reducing the time required to complete the survey, such that this ranged from 2 min to as much as 40+ min. The full details of the participant selections are included in the Supplementary Material.

In general, the criteria selected by the fewest participants were ranked lowest in the results. An interesting example of this is Noise Levels in the Environment theme—only 7% of participants selected Noise Levels as of key importance, which may have led to the low weighting score of 1.5%. This may be an anomaly, due to the concept that people are not used to thinking about noise in an energy project, or that there is a perception that existing noise control regulations are sufficient. Again, further work is recommended involving semi-structured interviews to test and confirm the most appropriate weightings for the Data Management themes, alongside investigation and consultation with appropriate experts to confirm whether the resulting weights for indicators with very low response rates are appropriate. This will provide additional evidence for stakeholder and expert opinions regarding which assessment themes and criteria are important to understand the benefits and barriers of SLES project development. Subsequently, case study analyses will be carried out using the EnergyREV MCA-SLES tool to confirm the effectiveness of these weights.

Table 4. Alignment of KPI themes and criteria with the existing literature.

Articles	KPI Theme (Number of Criteria)
This study	Data Management (6)	Technical Performance (5)	Business & Economics (6)		Environment (6)	People & Living (7)	Governance (6)
Heo et al. [5]	Technological (4)		Market (3)	Economic (3)	Environmental (3)		Policy (4)
Kaya and Kahraman [46]	Technical (7)		Economics (9)		Environmental (9)	Social (4)
Daim et al. [23]	Technical (6)		Economic (3)		Environmental (3)	Social (1)
Štreimikienė et al. [6]	Technological (4)		Economical (4)		Environment protection (4)	Social ethics (3)	Institutional & political (5)
Sahabuddin and Khan [11]			Economics (3)		Environmental (3)	Social (6)
Barney et al. [8]	Technical (2)		Economics (2)		Environmental (2)	Social (2)

shows that the six KPI themes selected in this study are broadly aligned with previous research in this area. Similarly, the number of criteria within these themes is broadly aligned to, or exceeds those in similar work. These themes and criteria represent key social prosperity, environmental, economic, and technological factors crucial for a successful SLES project.

4. Conclusions and Policy Implications

The work presented in this article provided insight into the priority weighting for criteria that will be used in a multi-criteria assessment tool being developed for Smart Local Energy Systems. This EnergyREV MCA-SLES tool is designed to examine the performance and benefits of SLES projects across a comprehensive set of KPIs (or criteria) grouped into six thematic areas (Technical Performance, Data Management, Governance, People and Living, Business and Economics and Environment). These KPIs and themes were identified through an extensive literature review and refined through stakeholder consultation in previous research. A discrete choice experiment was carried out to identify stakeholder views on the weightings for these KPIs and themes.

The DCE consisted of a main survey that focused on comparing the six KPI themes and an additional six optional surveys to independently assess the detailed indicators within each theme. These surveys asked each stakeholder to answer a series of simple pairwise questions comparing alternative hypothetical SLES in order to indirectly reveal their preferences. The results provide a set of weights for the six themes which will be used to develop an overall score in the EnergyREV MCA-SLES tool. It was revealed that the themes regarding environmental impact, people and living conditions were generally considered the most important. In contrast, data management was considered the least important.

There does, however, remain some uncertainty for the KPIs that received very few responses—either because the survey had too few participants or the KPI itself was not selected by the participant as a key criterion. This includes the Data Management theme, where only 24 respondents completed the survey, or Noise Levels within the Environment theme, which was only selected by four participants. Additional work involving semi-structured interviews with selected field experts is recommended to confirm and test the validity of the results retrieved from this DCE before final application in the EnergyREV MCA-SLES tool.

Finally, an independent standardised assessment tool such as the EnergyREV MCA-SLES tool will help SLES project developers by providing a route-map and checklist to support replication which may be utilised to enhance investors’ confidence. In the long term, the tool can also assist policy makers to identify areas where policy change is needed to enable progress towards a sustainable energy transition.