Education for Sustainable Development in Higher Education Rankings: Challenges and Opportunities for Developing Internationally Comparable Indicators

Abstract

As more higher education institutions strive to embed sustainable development principles in their teaching, it becomes increasingly important to identify indicators that can measure institutional contribution in a meaningful and internationally comparable manner. This paper shows that existing sustainability rankings, such as the UI Green Metric and THE Impact ranking, have paid relatively little attention to indicators on Education for Sustainable Development (ESD). In a quest to develop such indicators for U-Multirank—the multi-dimensional transparency and ranking tool—we reviewed the literature, consulted experts, and ran a survey amongst practitioners. This article summarises opportunities and challenges for developing internationally comparable ESD indicators in the higher education sector, discussing indicator relevance, validity, and feasibility. The results suggest that (i) ESD indicators are considered highly relevant by diverse stakeholders; (ii) the majority of HEIs surveyed are planning to collect ESD data within 3 years, signalling good prospects for data feasibility; (iii) the ESD indicators proposed so far still lack criteria that would allow one to sufficiently identify and compare these indicators across countries, inhibiting indicator validity. At least three potential definitions are used by HEIs. The results of this paper can contribute to the discussion on identifying appropriate criteria for the development of ESD indicators and their use in international rankings.

1. Introduction

Problem Statement and Rationale for ESD Indicators

Responding to urgent policy priorities [1,2,3] and societal concerns [4,5,6], a steeply growing number of higher education institutions (HEIs) have embarked on a journey to embed sustainable development in their key functions. For example, in 2021, more than 1100 institutions from 94 countries participated in the Times Higher Education (THE) Impact Ranking to show their contributions to the Sustainable Development Goals (SDGs) [7]. However, the pace at which sustainability transformations take place at HEIs varies considerably depending on the local context and capacity of each institution [8]. Internationally comparable yet locally meaningful indicators can support HEIs in learning from each other and assessing their progress.

While transformation towards sustainability in the higher education (HE) sector requires a systematic approach across all functions and the HE ecosystem [9], this article zooms in on one promising area—education and, particularly, Education for Sustainable Development (ESD). Education prepares (or underprepares) today’s students to be the decision makers of tomorrow, who must deal with the current and future sustainability challenges [10,11]. ESD supports students in developing critical competencies, such as systems thinking and integrated problem solving to address these challenges [12,13].

A substantial body of literature on sustainability indicators already exists [14,15,16,17,18,19,20], yet widespread use of indicators is hindered by uncoordinated practices and a lack of consensus on appropriate frameworks, methodologies, and impact assessments [19,21]. Numerous researchers have proposed that stronger harmonisation is needed at a global scale [21,22,23,24]. Cooperative action is required to monitor and achieve goals and reduce inefficiencies. Moreover, a widely shared and accepted knowledge base may better inform sustainability policies and create synergies across global efforts [24]. At the same time, the indicators must remain relevant for the local context [16,25]. The objective of this article is to explore promising ESD indicators for the HE sector by learning from existing rankings, such as the UI Green Metric and THE Impact ranking.

This paper aims to address three research questions. First, it explores the extent to which ESD-related indicators are available in the existing higher education sustainability rankings. Second, it identifies challenges for developing internationally comparable ESD indicators. Lastly, it reviews and proposes the most promising ESD indicators against three commonly found criteria in the literature on indicator assessment—relevance, validity, and feasibility [26,27,28]. The criteria are discussed in detail in Section 3.7. The following questions are proposed:

• RQ1: To what extent are ESD-related indicators available in the existing higher education sustainability rankings?
• RQ2: What are the challenges for developing internationally comparable ESD indicators in the higher education sector?
• RQ3: What are the most promising ESD indicators in the higher education sector, assessed against relevance, validity, and feasibility?

This research was undertaken as part of the U-Multirank project. U-Multirank is a multi-dimensional ranking and transparency tool featuring over 1900 HEIs [29], aiming to provide relevant and user-driven comparisons [30]. Recognising that the needs and priorities of U-Multirank participants are shifting, in 2019, a stakeholder consultation process was started to identify promising ESD indicators to include in U-Multirank. The author of this article is a member of the U-Multirank team for identifying and proposing the indicators.

2. Relevance of Sustainability in Higher Education

2.1. ESD in Global and European Policy Priorities

To determine the relevance of sustainable development indicators, we first reviewed recent policy documents. Sustainable development in policy discourse is commonly defined as “the ability to […] meet[s] the needs of the present without compromising the ability of future generations to meet their own needs” [31] (p. 16). Over the years, the concept has evolved to reflect three dimensions of sustainable development—environmental, social, and economic [1]. Although highly relevant today, sustainable development is not a novel policy priority. Already in 1987, an urgent call to action was made “to propose long-term environmental strategies for achieving sustainable development by the year 2000 and beyond” [31] (p. 5). More recently, the Agenda 2030 for Sustainable Development placed these aspirations in clear focus with 17 Sustainable Development Goals (SDGs) [1], mobilising stakeholders from all regions and diverse fields. As an essential element of transformations towards sustainable development, ESD is gaining increased recognition globally [32].

The European higher education policy documents and initiatives signal a clear priority given to sustainable development for the upcoming decade (2020–2030). Within the European Union (EU), education is seen as a critical component for building the capabilities of the EU’s citizens to attain sustainable development goals. The European Commission (EC) envisions that European education institutions at all levels should embrace the SDGs, transforming into organisations where skills for sustainability are both taught and practised. To enact the vision, reforms ranging from building green campuses to adjustments in the curriculum have been proposed [33]. The Council of the European Union [34] has recommended that, as part of the revised ‘key competences for lifelong learning’, all member states should mainstream sustainability education, including ESD, across entire education levels. As part of the European Green Deal, which outlines a long-term strategy for 2050, the Commission intends to support the development of a European competence framework to “assess knowledge, skills and attitudes on climate change and sustainable development” [2] (p. 19) and support teacher-training programs. One of the most notable EU initiatives in recent years is the European Universities Initiative. Launched in 2018 by the EC, it has financed 41 alliances across Europe to tackle “big issues facing Europe (such as climate protection, democracy, health, big data, migration)” [35] (p. 1). Many initiatives take place at a national level. In Finland, a forum for sustainable development in higher education has been established [36].

2.2. Education for Sustainable Development over Time

Since Education for Sustainable Development (ESD) is used as a key reference framework for indicator development in this paper, a brief overview of ESD and its development over time is provided in this section. ESD is a holistic learning approach that “empowers learners to take informed decisions and responsible actions for environmental integrity, economic viability and a just society, for present and future generations, while respecting cultural diversity” [37] (p. 4). First institutionalised in 1992 by the United Nations Educational, Scientific, and Cultural Organisation (UNESCO) [16], it is now recognised in key policy documents, including the 2030 Agenda for Sustainable Development (SDG 4.7), the Paris Agreement [32], and the EU higher education policy on key competencies [34]. ESD is also directly linked to the 17 SDGs [32] and aligned with the 2030 Agenda for Sustainable Development [38].

While ESD as a concept was first promoted by UNESCO in 1992, wider recognition of ESD increased during the UN Decade on Education for Sustainable Development (DESD, 2005–2014), followed by the Global Action Programme on ESD (GAP, 2014–2019). Both programs aimed to integrate sustainable development principles into education and learning [16], with the GAP program scaling up DESD achievements [32]. The most recent follow-up program is ESD for 2030, which “aims to build a more just and sustainable world through strengthening ESD and contributing to the achievement of the 17 SDGs” [38] (p. 14). Prior to ESD for 2030, UNESCO published a report titled ‘Education for Sustainable Development Goals: learning objectives’, linking ESD to all 17 SDGs [32].

Given ESD’s (i) direct links to sustainable development [32,38], (ii) continued presence and growing international recognition in policy, practice, and academia over time [12,39,40,41,42], and (iii) prescriptive yet non-restrictive definition [32] (p. 7), it can serve as a valuable reference framework for building international consensus on education indicators centred around sustainable development.

2.3. Higher Education Sustainability Rankings

Higher education rankings have been widely researched and thoroughly criticised [43,44,45,46,47], yet have remained relatively popular. While a large number of assessment tools exist for sustainability and sustainable development in the higher education sector (e.g., AISHE 2.0, STARS, GASU, ASSC, PSIR, SAQ, SustainTool, UniSAF) [14,15,16,18,19,20], only a few international rankings so far have undertaken the task—most notably, the UI Green Metric and THE Impact Ranking. A handful of articles have reviewed the approaches used in these rankings [48,49,50,51], and several recognised the importance of ESD, but did not specifically focus on ESD. Other sustainability rankings likely exist, particularly at the national level. One such example is a Dutch student-led sustainability ranking—Sustainabul. Thus, other researchers are welcome to identify new sustainability rankings and further contribute to the discussion.

3. Methodology

3.1. Conceptualising Sustainability in Key HE Functions

Sustainability in higher education can be embedded in all key functions. According to the United Nations guidelines, an integrative, whole-school/institution approach to embedding sustainability in HEIs is preferred. Such an approach often distinguishes six major dimensions that vary slightly in their formulation: (i) education (also ‘teaching and curriculum’, (ii) research, (iii) societal engagement (also ‘external community’, ‘community outreach’), (iv) campus operations (also ‘facilities’, ‘operations’) (v) organisational management (also ‘governance’, ‘leadership’), and (vi) assessment and communication (also ‘assessment and reporting’, ‘monitoring’). In this article, we refer to six such functions, as indicated in Figure 1, while recognising that multiple naming conventions are possible and have evolved over time [15,52,53].

For example, initially, Cortese [52] proposed four dimensions: education, research, campus operations, and community outreach. Soon after, Lozano [53] added ‘assessment and reporting’. More recently, Caeiro [15] used ‘external community’ instead of ‘community outreach’ and ‘assessment and communication’ in place of ‘assessment and reporting’. In this paper, we use ‘societal engagement’ instead of ‘external community’ to indicate mutually beneficial relationships between higher education institutions and wider society [54]. Finally, in recent years, more research has been done on the role of leadership [55,56,57], governance [18,58], and organisational change and transformation [59,60,61,62] in promoting sustainability in higher education. Since change can come from top-down, bottom-up, and middle-out approaches, organisational management is more encompassing than the other two and aligns with Caeiro’s suggestion [15]. As Lozano [53] notes, these dimensions are interdependent.

This paper focuses on sustainability through the function of education while also touching upon assessment and communication. This limitation should be acknowledged. It is our intention to further elaborate on the remaining dimensions in future.

3.2. Conceptualizing ESD Dimensions in Education

When developing indicators on education quality, it is common to distinguish between input, process, and output variables, which are sometimes called presage, process, and product variables, respectively [63,64]. Input variables exist within a university context prior to students beginning their studies, and they include resources, degree of student selectivity, the quality of students and staff. Process variables characterise the teaching and learning process, including measures of student engagement, pedagogies employed, teacher training, and learning environment. Output variables focus on student outcomes, including grades, student retention, and employment. Out of the three categories, process variables are shown to be the best predictors of the learning gain, or how much students learn at HEIs [63].

ESD is increasingly considered “an integral part of quality education” and “encompasses learning content and outcomes, pedagogy and the learning environment” [38] (p. 8). While mapping of the variables to three categories is not always straightforward [63], Figure 2 depicts how the ESD dimensions could be linked to the three commonly used dimensions for education quality—input, process, and output. The ESD dimensions cover some, but not all variables for education quality.

When aligning the education quality and ESD dimensions, ESD’s learning content falls under input variables, since curricula and the corresponding learning content are often prepared prior to the teaching process. In ESD, learning content is addressed by integrating critical issues such as climate change and poverty into the learning curriculum. More recently, topics linked to 17 SDGs have been emphasised. ESD’s dimensions on pedagogy and the learning environment are mapped to process variables, as these dimensions directly affect the learning process and educational gain. In ESD, these dimensions are addressed by utilising interactive, learner-centred pedagogy and providing a whole-institution approach that would enable learners to “live what they learn and learn what they live” [38] (p. 8). Finally, learning outcomes are mapped to output variables. In the ESD approach, learning outcomes are designed to empower learners to take responsibility for the present and future generations and are linked to eight key competencies, including systems thinking, anticipatory thinking, and integrated problem solving [32].

As part of U-Multirank’s new indicator development process, it was decided to focus on three out of the four dimensions, eliminating the ‘learning environment’. Although highly relevant, it was considered too novel and vague to be operationalised in specific indicators in the short-term future.

3.3. Overview of the Research Process

The research used an exploratory sequential mixed-method design [65], and it was carried out in three consecutive phases. First, an exploratory literature review was conducted to identify the relevance of sustainability in the higher education sector. Resources were gathered from the academic literature, recent policy documents, and higher education sustainability ranking and assessment tools (e.g., THE Impact Ranking, UI Green Metric, UniSAF). The insights obtained helped to prepare for the second phase and are summarized in Section 2. In the second phase, two semi-structured focus groups were organised in an online setting to collect qualitative feedback (n = 13). The aim of the focus groups was to discuss indicator relevance, validity, and feasibility with international experts, practitioners, and student representatives. In addition, an open consultation was held with the U-Multirank advisory board, where they were invited to comment on the proposed indicators. The insights from the semi-structured focus groups and the advisory board meeting informed the design of the quantitative indicator survey utilised in the third phase. The survey was distributed to the U-Multirank participants to obtain quantitative data on indicator validity (n = 227) and feasibility (n = 256). An overview of the stakeholders consulted is displayed in

Table 1. Overview of the stakeholders consulted.

Stakeholders	Mode of Consultation	Nr. of Participants	Gender	Geographic Representation	Type of Organization
Experts and practitioners	Semi-structured focus group	6	50% female, 50% male	Europe, North America	Higher Education Institutions, Intergovernmental Organisations (UNESCO, Green Office Movement, Aurora Network, Green Hub)
U-Multirank Advisory Board	Open format consultation	16	50% female, 50% male	Worldwide Europe- centred	Higher Educations Institutions, Intergovernmental Organisations (EUA, EURASHE, OECD, ESU, CEASER, ESN, IAU)
Student representatives	Semi-structured focus group	7	57% female, 47% male	Europe	Student organisations (ESU, ESN)
U-Multirank participants	Survey	227 256	Not available	Worldwide	Higher Education Institutions

below.

3.4. Rankings and the Assessment Tool Reviewed

To identify existing indicators that focus on either education for sustainable development (ESD) or education linked to SDGs or sustainability, we reviewed rankings that are specifically focused on sustainability in the higher education sector. As a reference point, one holistic assessment tool was also included. The rankings were selected using a convenience sample based on their (i) focus on sustainability, (ii) visibility in the higher education sector, and (iii) potential relevance for U-Multirank. The rankings reviewed were: the Impact Ranking, UI Green Metric, Sustainabul, and UniSAF framework. The UniSAF framework is a holistic assessment tool that is used as a reference due to the relatively high number of education indicators. While the list is not exhaustive and cannot claim to be representative of all sustainability rankings, and particularly not assessment tools, it highlights some of the more recent indicator trends in sustainability or SDG-related rankings. As a result of the indicator review, a preliminary list of potential new ESD indicators was compiled. This list was used as a starting point for stakeholder consultations.

3.5. Focus Groups

Between September 2020 and April 2021, the U-Multirank project team consulted a diverse group of stakeholders to evaluate the need for internationally comparable indicators on ESD. Amongst the participants were experts, practitioners, policymakers, and student representatives. Three separate consultations were carried out. First, an expert panel on Education for Sustainable Development (6 participants) provided comprehensive feedback on the relevance, validity, and feasibility of indicators. Next, the U-Multirank Advisory Board (16 participants) was invited to provide general feedback on the relevance of the indicators in an open consultation format. A wide spectrum of representatives was included in these consultations (e.g., representatives from UNESCO, AURORA network, OECD, IAU, and EUA). Lastly, a separate consultation was set up with student representatives from the European Student Union (ESU) and Erasmus Student Network (ESN) (7 participants) to obtain student perspectives on indicator relevance, yet students also shared some feedback on indicator validity and feasibility. The final report, including meeting notes, is publicly available.

While the stakeholders consulted represented several international institutions (e.g., UNESCO, OECD, IAU), the majority of the participants were engaged in European institutions, initiatives, or networks. Therefore, the insights obtained during the consultation process would be of higher importance to European countries and, to a certain extent, North America. No stakeholders from Africa, Asia, Australia, or South America were consulted. For a brief overview of the geographic coverage of stakeholders consulted, see Table 1.

3.6. Practitioners Surveyed

In order to identify ESD indicators with the highest feasibility (availability of data) and the most commonly used definitions for ESD or sustainability-related education, the U-Multirank team conducted a survey amongst HE practitioners. The survey was sent out to U-Multirank participants between February and June 2021. It was permitted to forward the survey to other colleagues at one’s institution, resulting in a larger number of participants than institutions. In the survey, neither of the two questions were required, which led to a different number of answers per question. The question on indicator data availability was completed by 256 respondents, while the question on the most commonly used ESD definitions was answered by 227 respondents. Respondents held various positions, including ranking officers, sustainability coordinators, and international affairs coordinators.

3.7. Criteria for Indicator Assessment

Building on the insights from the academic literature, policy papers, and existing rankings, a preliminary list of indicators was prepared for stakeholder consultations. During the focus groups, stakeholders were invited to assess indicators on three criteria: relevance, validity, and feasibility. Such criteria are frequently used in the research literature on education indicators [26,27,28,66]. The same criteria were used to assess other new indicator groups for U-Multirank, such as social inclusion [66] and effective teaching and learning. The relevance criterion was seen as critical, since indicators would only be used if the phenomenon measured is considered important by the users [27]. If the relevance criterion is not met, institutional leaders and practitioners will not invest time in data collection. The following criterion—validity—evaluates if an indicator reflects the phenomenon it is meant to represent [67]. Validity is particularly important for ESD indicators, since no operational definition is available at an international level, and multiple different definitions are used, a result indicated by the feasibility survey. The third criterion, feasibility, addresses practical aspects of the data collection process and data availability, including institutions’ capacity and readiness to obtain the required data considering their available resources and expertise [28].

4. Results

4.1. Rankings Reviewed (RQ1)

Section 4.1 aims to answer the first research question: (RQ1) To what extent are ESD-related indicators available in the existing higher education sustainability rankings? The rankings reviewed were the Impact Ranking, UI Green Metric, and Sustainabul. In addition, UniSAF, a holistic assessment tool, was included as a reference due to the large number of education indicators it features. Below,

Table 2. An overview of the sustainability ranking and assessment tools.

Category	THE Impact	UI Green Metric	Sustainabul	UniSAf
Focus	SDGs	Environmental Sustainability	Student-driven assessment	Holistic sustainability self-assessment
Level	Institutional	Institutional	Institutional	Institutional
Coverage	International	International	Dutch	International, EU-oriented
Release year	2019	2010	2012	Not applicable
Participants (2020)	768	912	30	Not applicable
Areas covered	17 SDGs in the areas of (1) research, (2) stewardship, (3) outreach, (4) teaching,	(1) energy and climate change (2) education and research (3) transportation (4) waste (5) setting and infrastructure (6) water	(1) education, (2) research, (3) operations,	(1) education, (2) research, (3) community, (4) operations, (5) governance
Indicators: Learning Content	(1) local education programs on climate change/SDGs; (2) community outreach programs	(1) ratio of sustainability courses to total courses/subjects	(1) sustainability-oriented education programs (2) sustainability-oriented education minors (3) SDGs in education curriculum	(E-1) courses focused on sustainability (E-2) percentage of courses (E-3) availability of courses to students (E-4) educational offerings for general public (E-5) sustainability specialisation (E-6) sustainability focused service learning (E-7) student enrolment
Indicators: Pedagogy			(4) training and support for teaching staff	(E-11) professional development of sustainability educators (E-12) links between sustainability research and education (E-13) partnerships (E-8) student involvement
Indicators: Learning Outcomes	(3) graduates in relevant fields, such as % of health graduates (SDG3)			(E-9) alumni destinations (E-10) educational outcomes

provides a comparison of all four tools.

4.1.1. Indicators in the THE Impact Ranking

The Times Higher Education (THE) Impact ranking, first released in 2019, claims to be the only ranking to assess the performance of higher education institutions against the Sustainable Development Goals as defined in the Agenda 2030 for Sustainable Development. The participation in the ranking is voluntary, its coverage is global, and assessment is done at the institutional level across the SDGs, as well as separately on each SDG. To participate, institutions need to submit information on at least three SDGs in addition to SDG17 (strong institutions), which is mandatory [68].

The second edition (2020) included 768 HEIs from 85 countries [69]. SDGs were assessed across four areas—(1) research, (2) stewardship, (3) outreach, and (4) teaching. The THE Impact ranking collects information about sustainability-related education under two goals: SDG13—Climate Action and SDG17—Partnerships for the Goals. Under SDG 13, HEIs need to submit information about education programs on climate action, and for SDG 17, education must be centred around the SDGs [70,71]. Several other metrics are related to sustainable development, but extend beyond the scope of education. For example, SDG4 (Quality education) [72] reports on inclusive access, while several other SDGs (SDGs 6, 7, 14) consider outreach programs to local communities ranging from topics about good water management (SDG6) [73], […] energy efficiency and clean energy (SDG7) [74], and sustainable management of fisheries […] [75] (SDG 14).

4.1.2. Indicators in the UI Green Metric

The UI Green Metric World University Ranking, launched by Universities Indonesia in 2010, aims to provide information about the sustainability of universities around the world. Participation in the ranking is voluntary, its coverage is global, and assessment is done at the institutional level. Data collection happens through online questionnaires sent to university administrators [76,77].

The ranking criteria are built around six pillars—energy and climate change (21%), education and research (18%), transportation (18%), waste (18%), setting and infrastructure (15%), and water (10%). The assignment of weights suggests that the ranking is skewed towards operational measures (82%), utilising a wide range of criteria, such as the campus location, amount of green space, energy use, transport, water use, recycling, and waste treatment. Under the ”Education and Research” pillar, one out of seven indicators is dedicated to education—”ratio of sustainability courses to total courses/subjects” [78].

4.1.3. Indicators in Sustainabul

Sustainabul is a Dutch national sustainability ranking of higher education institutions (HEIs). First released in 2012, Sustainabul is an example of a bottom-up ranking, commenced and run annually by a national student network, ”Studenten voor Morgen” (Eng: Students for tomorrow). ”Studenten voor morgen” created the ranking to encourage institutions to become more sustainable through competition and knowledge sharing. Sustainabul assesses HEIs on three dimensions—education, research, and operational management—and gathers information about the best sustainability practices [79].

In 2020, Sustainabul featured the thirty largest Dutch HEIs in terms of student enrolments, covering both universities and universities of applied sciences. Sustainabul’s methodology allocates equal weights to three key dimensions—sustainability in education, research, and operational management (maximum of 110 points per category)—while best practices can receive an additional 30 points. The education dimension consists of four measures: education programs centred on sustainability (50%), minors centred on sustainability (30%), training and support for teaching staff (15%), and integration of SDGs in the curriculum (5%) [80].

4.1.4. Indicators in the UniSAF Framework (Green Office Movement)

The Green Office Movement was launched in 2010, aiming to create a sustainability platform that empowers students and staff to embed sustainability in the curriculum, research, operations, community, and governance of their higher education institution [81].

One of the resources provided by the Green Office Movement is University Sustainability Assessment Framework (UniSAF). Although not a ranking, the UniSAf framework offers a broad spectrum of indicators for a holistic assessment of an institution. The indicator categories include education, research, community, operations, and governance. Thirteen indicators have been proposed for education, and these are split into three categories—educational offering, students, and course quality [82].

4.1.5. ESD Indicators in the Existing Sustainability Rankings and Assessment Tool

An overview of the indicators in the HE sustainability rankings allows us to answer the first question: To what extent are ESD-related indicators available in the existing higher education sustainability rankings? All three rankings have indicators on learning content—namely, programs, minors, courses, or community outreach focused on climate change, sustainability, or SDG topics. Only one of the rankings, Sustainabul, has indicators on pedagogy, referring to training and support provided for teaching staff. In addition, for learning outcomes, only one of the rankings provides an indicator—the number of health graduates.

On the other hand, the reference tool UniSAF has multiple indicators in each category. Learning content also considers not only courses, but also the availability of courses to the student population (e.g., access in terms of regulations, timing, pre-requisites) and service-learning courses. For pedagogy (course quality in UniSAf), UniSAF considers professional development and student involvement (the extent to which students feel that they can shape their learning experiences) and links to research and partnerships with external parties. Finally, for learning outcomes, UniSAF considers educational outcomes based on the sustainability competencies acquired and the alumni destinations.

Thus, it appears that the existing rankings provide little input for ESD indicators, mostly emphasising learning content, but with somewhat limited attention to pedagogies and learning outcomes. However, frameworks such as UniSAF are helpful in developing new ESD indicators, since they consider learning content, student involvement, professional development of teachers, sustainability competencies, and alumni destinations, effectively addressing most ESD dimensions (see Figure 2).

4.2. Challenges when Developing ESD Indicators (RQ2)

Section 4.2 aims to answer the second research question: (RQ2) What are the challenges for developing internationally comparable ESD indicators in the higher education sector? Building on the indicators found in the aforementioned rankings and, particularly, the UniSAf framework, a preliminary list of ESD indicators was developed for stakeholder consultations. During this stage, potential challenges for internationally comparable indicators were identified.

An overview of the potential indicators, their operationalisation, and challenges is available in

Table 3. Preliminary ESD indicators and underlying challenges.

Indicator Category	Indicators	Challenge
Learning content	ESD course offering (% of total courses)	(1) lack of criteria to identify education that meets ESD standards; (2) limited ability to compare education workload internationally
	ESD minor offering (%of total minors)
	ESD program offering (% of total programs)
Pedagogy	ESD training for educators (% trained within last 5 years)	(3) lack of criteria to identify training that meets ESD standards; (4) limited ability to compare training workload internationally; (5) limited ability to recognise initial teacher training’s contribution to the ESD approach
Learning Outcomes	ESD course graduates (% of total course graduates)	(1) lack of criteria to identify education that meets ESD standards; (2) limited ability to compare education workload internationally
	ESD minor graduates (% of total minor graduates)
	ESD program graduates (% of total program graduates)
	ESD competencies (alumni self-reported scores)	(6) lack of criteria to assess ESD competencies (eight official ESD competencies identified by UNESCO)
	ESD alumni in relevant SD-related jobs (% of total alumni)	(7) lack of criteria to identify SD-related jobs

. Indicators were grouped around three ESD dimensions—learning content, pedagogy, and learning outcomes. Under learning outcomes, graduate outcomes were also included, though these are technically not learning outcomes, but a broader outcome measure. In addition, a preliminary list of challenges for operationalising these indicators was identified, from which most were rooted in three main challenges. The first challenge stems from the need to agree on thresholds or guidelines to identify and, consequently, create a classification mechanism for ESD-related education/teacher training/jobs/competencies (see Table 3, challenges 1, 3, 6, 7). Since ESD is a holistic framework that encompasses multiple aspects of education, it is challenging to propose a specific yet contextually relevant definition that would allow the classification of educational offerings as ESD. Secondly, comparing workloads across different systems on a global scale is difficult (see Table 3, challenges 2, 4). At the EU level, the European Credit Transfer System (ECTS) can be used, but a conversion mechanism needs to be established at a global level. Lastly, national qualifications for teacher training differ substantially across countries. It is possible that in some systems, teachers are better prepared to teach ESD topics and teaching methods and, therefore, should not be penalised for not having additional training afterwards (see Table 3, challenge 5). This challenge was identified during the consultations, but has been included in this table for a better overview. This list is the first attempt to identify challenges and add to the existing literature. Any additions and modifications are welcome in future research.

4.3. Results from the Stakeholder Consultations and Feasibility Survey (RQ3)

Section 4.3 aims to answer the third research question: (RQ3) What are the most promising ESD indicators in the higher education sector, assessed against relevance, validity, and feasibility? To answer the question, the outcomes of stakeholder consultations are discussed, summarising reflections on the proposed ESD indicators’ relevance, validity, and feasibility. The procedure for stakeholder consultations consisted of focus groups and a survey, and it is described in Section 3.5 and Section 3.6.

4.3.1. Relevance

Indicator relevance is one of the critical factors when developing new indicators. While the policy documents signalled clear priority for sustainability indicators, additional consultations experts, practitioners, and students provided contextualised insights on the potential use and importance of indicators. During the stakeholder consultations on indicator relevance, three questions emerged: (i) Are ESD indicators relevant? (ii) Relevant for what purpose? (iii) Relevant for whom?

All stakeholders agreed that ESD and, more broadly, education linked to sustainable development are highly relevant and critical to attaining global policy goals. Therefore, (i) ESD indicators are relevant, since they can help to establish a baseline, reflect progress over time, and allow one to learn from others by providing a contextualised comparison. At the same time, indicators are imperfect measures and ‘what gets measured, gets noticed’. Therefore, all stakeholders should remain critical and keep examining the purpose of the indicators and parties involved.

Stakeholders noted that ESD indicators should (ii) fit the purpose, stimulating institutional learning and transparency rather than competition amongst HEIs. Higher education rankings typically create tension and competition and, therefore, are not well suited for new ESD indicators. For example, in a ranking, a higher education institution may drop in a rank even if it has improved its sustainability performance if the participant pool has changed from one year to another. Instead, ratings with transparent criteria and predetermined levels may be used. Moreover, the indicators should be (ii) holistic rather than focusing on education only, covering other functions, such as operations and governance. In addition, the indicators should also (ii) reflect student and educator experiences.

Stakeholders emphasised that indicators should be relevant (iii) for HEIs with diverse profiles, including universities of applied sciences and vocational education institutions. Still today, many rankings predominantly focus on traditional research universities. The new ESD indicators should reflect the needs and indicators relevant for all institutions, not favouring research-intensive institutions. In addition, (iii) national context should be considered as much as possible when developing indicators to make them relevant for a large number of countries and geographic regions. For example, while all ESD indicators were generally seen as relevant, it was noted that some national systems do not use a minor system, and such indicators would not be relevant in their system.

4.3.2. Validity

In addition to being relevant, indicators must be valid, representing the phenomenon that they aim to capture. Since the definition of ESD is sufficiently abstract, there is no one manner in which to operationalise ESD education. Therefore, a careful operationalisation of indicators is important. Such operationalisation requires agreeing on the criteria used to classify education as related to ESD. Since many HEIs focus on SDGs, yet are not familiar with ESD, we expanded the operationalisation to capture ESD education as well as SDG-related education. This led to the following question: (iv) What criteria can be used to classify education as ESD or SDG related?

In the case of U-Multirank, the selected criteria need to be sufficiently flexible to acknowledge efforts of diverse higher education institutions and geographic locations, yet they must also be internationally comparable. This creates a tension between flexibility and comparability, where an appropriate balance needs to be found.

During the stakeholder consultations, three classification criteria were proposed, yet a consensus on the preferred approach was not reached. The first focused on the (i) content covering complex problems from three perspectives (social, environmental, and economic), while the second emphasised the need for (ii) action-oriented teaching methods typical of the ESD approach. The last addressed (iii) SDG perspectives in a specific field. To investigate what criteria are used in practice, the U-Multirank team ran a survey, proposing the three potential definitions, as well as allowing respondents to add their own answers. Respondents could select multiple answers. As can be seen from Figure 3, in total, 227 respondents provided their answers, with many selecting multiple definitions. The most common ESD definition focused on complex problems considering three perspectives (36%), but alternative options, such as definitions linked to teaching methods (31%) and field-specific focus on SDGs (29%), were also commonly used. Amongst the 4% who mentioned other answers, the most common alternative was using learning goals that are often linked to key competencies (e.g., systems thinking, critical thinking) as a criterion for ESD or SDG-related education, while others mentioned extracurricular activities, such as learning with communities.

Validity is important, and for ESD or SDG-related education, establishing consensus on how to operationalise these indicators is essential. It should provide sufficient flexibility while also ensuring transparency and contextual relevance. The survey results indicate that all three definitions are seen as helpful, while the most common one is built on the commonly used definition of sustainable development, addressing complex problems across three dimensions—social, economic, and environmental.

During the stakeholder consultation, we also received feedback on other indicators. In particular, the validity of an indicator on teacher training was criticised. Teacher qualifications differ considerably across countries. Since some might entail elements of ESD and transformational teaching methods while others may not, the starting points of the educators differ. Hence, an indicator on ESD training may not represent the quality of ESD teaching. However, it was suggested that an indicator on ESD competencies, particularly those assessed by students, could serve as a good representation of learning outcomes. Overall, consultations showed that operationalising ESD- or SDG-related indicators is challenging. However, in the long run, building international consensus on the relevant criteria may facilitate the process.

4.3.3. Feasibility

Indicator feasibility is critical in determining whether indicators are eventually used. If data are not available or the cost of collecting the required data elements is too high for an institution, even highly relevant and potentially valid indicators may not be used. In order to understand the feasibility of the proposed indicators, we ran a survey, aiming to understand (v) what indicators have the highest feasibility scores?

The survey was shared with representatives from HEIs participating in U-Multirank. The representatives were asked whether they were collecting data on proposed ESD or SDG-related education measures. In this context, ‘SDG-related measures’ refer to SDGs and content linked to SDGs with a more holistic approach (teaching methods, ESD competencies). As shown in Figure 4, respondents could indicate if data were already collected or would be collected in the near future (within 1–3 years). The results were ordered based on the likelihood that the data would be readily available within three years. When we refer to the likelihood, we combine values for already available data with the data that shall become available in the near future. The total number of respondents answering the question was 256. However, the answers were not forced and varied slightly for each indicator, ranging from 229 to 247.

As can be seen from Figure 4, the four most common indicators, judged on the current and near-future data availability, are ”ESD courses offered” (89%), ”ESD-related programs” (80%), ESD course graduates (78%), and ESD program graduates (76%), featuring two indicators from the learning content category and two from the learning outcome category. For these indicators, more than 30% of the respondents indicated that the data were already available, while at least 40% reported that they would be able to provide it in the near future. For the remaining indicators, between 20 and 27% of the respondents were able to provide data right away, yet between 40 and 51% indicated that such data would be available in the future. The indicator with the lowest feasibility was ESD alumni in relevant jobs (64%), but even for this indicator, the majority of respondents expected to have data in the near future. During the consultations, stakeholders provided qualitative feedback on indicators they considered less feasible in terms of obtaining the required data. For example, it was noted that, in some national systems, minor programs are not used, and therefore, it would neither be possible to obtain data on such indicators, nor it would be relevant. On the other hand, ‘alumni in relevant jobs’ was seen as highly relevant, yet less feasible. The most likely data source would be existing institutional alumni surveys, with little control over response rates and often predefined job sectors. With predefined sectors, it might not be possible to identify if alumni are working on sustainability-related topics. For example, some engineers might be heavily involved in addressing sustainability challenges, while others are not. Thus, an additional question might be needed to understand whether alumni are engaged in sustainability-related jobs. Nonetheless, the feasibility survey suggests that most HEIs are proactively working on obtaining ESD-related data for even more challenging indicators.

5. Discussion

5.1. Prioritising ESD Indicators

Insights from stakeholder consultations and a feasibility survey allowed us to assess indicators on their relevance, validity, and feasibility and to prioritise them in three tiers based on the overall scores (see

Table 4. Evaluating the new ESD indicators on their relevance, validity, and feasibility.

Category	Indicator	Relevance	Validity	Feasibility	Priority
Learning content	ESD course offering	High	Medium	High	1st tier
	ESD minor offering	Medium	Medium	Medium	2nd tier
	ESD program offering	High	Medium	High	1st tier
Pedagogy	ESD training for educators	High	Low	Medium	2nd tier
Learning Outcomes	ESD course graduates	High	Medium	High	1st tier
	ESD minor graduates	Medium	Medium	Medium	2nd tier
	ESD program graduates	High	Medium	High	1st tier
	ESD competencies	High	Medium	Medium	2nd tier
	ESD alumni in relevant jobs	High	Medium	Low	2nd tier

). More detailed feedback for each indicator is available in

Table 5. Detailed feedback on indicator relevance, validity, and feasibility.

Category	Indicator	Relevance	Validity	Feasibility
Learning content	ESD course offering	High relevance, showcases availability of courses, often to a broad audience	Criteria for identifying and comparing (workload) ESD-related education are required; criteria need to be sufficiently flexible to fit various contexts and geographies; at least three definitions are commonly used: (i) multidisciplinary sustainability content, (ii) field-specific sustainability content and applications, (iii) multidisciplinary sustainability content combined with ESD teaching methods.	High feasibility, over 1/3 (41%) already collect data, 89% will be ready within 3 years
	ESD minor offering	Medium relevance, showcases availability of minor/specialisation yet ‘minors’ are not used in all countries	See above (row 2, column 4)	Medium feasibility, less than 1/3 (27%) already collect data, 74% will be ready within 3 years.
	ESD programs offering	High relevance, showcases availability of full programs	See above (row 2, column 4)	High feasibility, over 1/3 (40%) already collect data, 80% will be ready within 3 years
Pedagogy	ESD training for educators	High relevance, reflects educators’ knowledge and competencies, impacts educational quality	See above (row 2, column 4). In addition, the validity of the indicator has been criticised for not considering national teacher training qualifications, since some systems pay more attention to ESD-related teaching methods (e.g., student-centred, transformational learning)	Medium feasibility, less than 1/3 (27%) already collect data, 72% will be ready within 3 years
Learning Outcomes	ESD course graduates	Medium relevance, showcases the number of beneficiaries from the course	See above (row 2, column 4)	High feasibility, over 1/3 (31%) already collect data, 78% will be ready within 3 years
	ESD minor graduates	Medium relevance, showcases the number of beneficiaries from the minor yet ‘minors’ are not used in all countries	See above (row 2, column 4)	Medium feasibility, less than 1/3 (23%) already collect data, 70% will be ready within 3 years.
	ESD program graduates	High relevance, showcases the number of beneficiaries from the program	See above (row 2, column 4)	High feasibility, over 1/3 (31%) already collect data, 76% will be ready within 3 years
	ESD competencies	High relevance, showcases to what extent students have acquired ESD competencies	Currently, 8 ESD competencies have been defined by UNESCO (e.g., systems thinking, anticipatory competency). However, criteria need to be established to assess these competencies	Medium feasibility, less than 1/3 (24%) already collect data, 75% will be ready within 3 years.
	ESD alumni in relevant jobs	High relevance, showcases to what extent students utilise their education in the labour market and create impact	Criteria for identifying jobs related to sustainable development are needed; most likely requires self-assessment due to the multi-faceted nature of jobs.	Low feasibility, less than 1/3 (20%) already collect data, less than 2/3 (64%) will be ready within 3 years.

. The feasibility of the indicators was assessed against the possibility of obtaining the data within three years rather than immediately. In the first tier, four indicators were included: ESD courses, ESD programs, ESD course graduates, and ESD program graduates. All four indicators were assessed as highly relevant by stakeholders, had medium validity due to the need to clarify definitions, and medium to high feasibility, as indicated by the survey. By establishing a consensus around definitions for ”ESD educational offering”, it would be possible to increase the perceived validity of the indicators, as well as to enhance the feasibility of data collection. The insights from the feasibility survey indicate that, in the following years, the feasibility assessment is likely to move from a medium-high to a high score.

Furthermore, in the second tier, five indicators were included—ESD minors and minor graduates, ESD competencies, ESD training for educators, and ESD alumni in relevant jobs. ESD minors and ESD minor graduates received a medium score on all three dimensions. This may be partially attributed to some educational systems not offering minors, as suggested in the stakeholder consultations. However, for institutions where such programs are offered, such indicators could still be relevant. Furthermore, ESD training for educators was considered a relevant indicator, yet its validity was criticised due to differences in national systems for teacher qualifications and the potential to signal mistrust towards educators. It received a medium feasibility score. ESD competencies were seen as a promising way to incorporate student views, but were criticised for their subjectivity. Moreover, while eight ESD competencies have been communicated [32], these competencies still need to be operationalised into measurable indicators. ”ESD alumni in relevant jobs” would require thresholds to establish what constitutes ESD-relevant jobs. While highly relevant, the biggest drawback of the indicator was its feasibility score, as shown by the survey results and indicated by stakeholders. Alumni surveys might suffer from job classification methods that do not reveal whether sustainability is embedded in the job role and would likely differ across countries and even institutions.

5.2. Limitations

To identify new indicators, we used ESD as a reference framework. A different framework would most likely yield an alternative set of indicators. Moreover, as indicated by the UniSAF indicator, several additional indicators could be included, such as “educational offerings for general public (E-4)” and “sustainability focused service learning (E-6)” [82] if the education of local community members would have been included in the scope.

Moreover, throughout the stakeholder consultations, we aimed to find indicators that could be internationally comparable. For institutions that look to measure their own progress, it is not always necessary to develop internationally comparable indicators. Even though international indicators can facilitate peer-to-peer learning and provide additional reference points, at times, local contexts and needs might be stronger drives for certain indicators [16].

Furthermore, we used a snowballing method to select the stakeholders consulted, including international experts. While it is a relatively common approach given the limited availability of international experts, some selection bias can be expected. Similarly, for the feasibility survey, self-selection bias exists. While the survey was sent to all U-Multirank participants (n > 1700), only a relatively small share completed the survey. It is likely that these participants were already more interested and enthusiastic about sustainability topics.

5.3. Future Research

In order to establish ESD indicators, further research could be carried out on the specific guidelines that could be utilised for establishing ESD indicators. Moreover, a repeated feasibility study could be conducted in the near future to evaluate whether institutions have started collecting data on ESD indicators, as proposed by the feasibility survey in this study. In addition, this study explored indicators on three out of four ESD dimensions, eliminating the ‘learning environment’, since it was considered too novel and broad. However, further research could attempt to measure this dimension [38]. More broadly, another strand of future research could investigate the role of academic leaders, such as rectors, deans, and heads of departments, in implementing and supporting initiatives for monitoring ESD or SDG-related education at their institutions and academic units.

6. Conclusions

Global societal trends (e.g., Extinction Rebellion), international policy discourse [1], and large-scale initiatives [35] signal a clear priority given to sustainable development initiatives. Higher education institutions have a critical role in contributing to these priorities, particularly by preparing future leaders in a diverse set of fields. Hence, these institutions should be transparent in communicating their efforts to existing and prospective students, as well as broader society. To identify internationally comparable ESD indicators, this paper reviewed the relevant literature and existing sustainability rankings, conducted stakeholder consultations, and ran a survey. The results of this paper are threefold. First, it compares existing higher education sustainability rankings—the Impact Ranking, UI Green Metric, Sustainabul—and highlights that these rankings pay limited attention to ESD indicators. In particular, only one out of three rankings provides measures on pedagogy and learning outcomes. Indicators on the available sustainability education (courses, minors, programs) are common across all three rankings. A holistic assessment framework, UniSAF, can provide valuable insights into the future development of ESD indicators, since it contains a much more comprehensive list of indicators, including measures on pedagogy and learning outcomes.

Second, building on the existing rankings and using ESD as a reference, the paper provides a preliminary list of indicators together with the underlying challenges. The challenges include a lack of criteria for identifying ESD-related education while recognising that such criteria need to be sufficiently flexible to fit diverse contexts and geographic locations. This challenge is not limited to educational offerings, and also includes ESD-related competencies, teacher training, and alumni job destinations. In addition, criteria for comparing workloads across different systems need to be established, which can be similar to the ECTS system used in Europe.

Third, stakeholder consultations and surveys provided insights on indicator relevance, validity, and feasibility. ESD indicators were considered highly relevant by all stakeholder groups (students, experts, and practitioners). However, assessing general relevance is not enough. The potential users of the indicators need to consider ‘for what purpose are indicators relevant?’ and ‘for whom are these indicators relevant’? The stakeholders believed that indicators should promote institutional learning and be inclusive of all types of institutions, not only research universities. The discussions on indicator validity revealed that a consensus needs to be established on appropriate criteria for identifying ESD-related courses/programs/minors/relevant alumni jobs and educator training. At least three types of definitions are available to operationalise ESD courses and programs—(i) content-driven knowledge linked to complex interdisciplinary challenges from social, economic, and environmental perspectives, (ii) teaching methods focused on transformational learning, and (iii) field-specific content knowledge of sustainability. Lastly, the survey on indicator feasibility revealed that, while less than half of the respondents currently collect information on any of the proposed ESD indicators, more than half (64%) plan to collect such information within three years, signalling strong interest in and commitment to monitoring this area.