Practice-Based Program Evaluation in Higher Education for Sustainability: A Student Participatory Approach

Abstract

While learning competencies in education for sustainable development are increasingly recognized as important, few empirical studies consider competencies delivered at a program level. The purpose of this paper is to demonstrate how a program evaluation can be approached through a participatory approach, and what this means for learning competencies for sustainability. The innovative method chosen was to implement a student-led evaluation of the program or a form of practice-based learning whereby students engaged in a participatory evaluation of their own program. This evaluation involved a mixed-methods research design and engaging with different actors—from other students and teachers in the program to alumni, administrators and employers. Students agreed on what competencies to evaluate against, then designed their assessment to gauge how and in what way five key competencies were being delivered. The program delivers competencies for sustainable development, yet there was some discrepancy between what students experienced in the program and what teachers believed to be delivering in the classroom. The learning-by-doing approach suggests that a sixth competency—implementation skills—is relevant to teaching for sustainable development. A participatory, student-led approach to evaluating a Master program is a novel contribution to the literature, which in itself led to the development of competencies for sustainability, particularly strategic, interpersonal and implementation skills.

1. Introduction

In 2017, UNESCO published a report titled Education for Sustainable Development Goals (SDGs): Learning Objectives, presenting each of the SDGs in relation to effective teaching methods towards achieving these goals. Since then, these ambitions have been reinforced through UNESCO’s Framework for the Implementation of Education for Sustainable Development (ESD), Beyond 2019 [1]. In higher education institutions, the SDGs, and sustainability more generally, have been gaining in importance at the level of courses but also teaching programs, including Master programs, as a new paradigm for learning opportunities [2,3,4,5]. Determining whether and how effectively (sustainability) teaching programs are equipping students to be(come) professionals able to affect positive change and continually add to their body of knowledge thus becomes an area of further study. Program evaluation “is a form of enquiry which describes the achievements of a given program, provides explanations for these, and sets out ways in which further development might be realized” [6] (p. 99). In other words, program evaluations seek to ascertain the quality and fitness for the purpose of a given (education) program and change it where needed. Institutions will regularly evaluate their programs for quality, effectiveness, and innovation, but evaluation at the program level specific to “sustainability learning outcomes” is less prevalent [7,8,9,10].

While ESD contributes to all SDGs, it is especially relevant for SDG4 on Education, and Target 4.7 in supporting the knowledge and skills needed for sustainable development. What to evaluate in relation to sustainability outcomes has been explored in the literature, leading to a growing interest in sustainability learning competencies, usually assessed at the level of course offerings [5,7,11,12,13,14,15]. Yet how to engage in an evaluation process remains understudied; that an “external” reviewer will evaluate a program is usually assumed. We present the results of practice-based Master student engagement towards self-evaluating [16] their own sustainable development program at the University of Geneva. Through participatory processes and mixed methods, students reflected on the learning offered by their program, seeking to assess what competencies for sustainability it delivers, and did so by engaging with different actors in their study, from teachers in the program to alumni and administrative staff. A second question arose during the evaluation process, namely, what do students gain by participating in the evaluation of their own program. This contribution seeks to discuss the benefits and limitations of participatory methods towards program evaluation and the implications for the development of skills and competencies for sustainability.

We begin by discussing the literature on learning competencies for sustainability by reflecting on what proposals have already been made towards integrating participative methods. We then discuss the methods used in our study, followed by an overview of the results. We end with a discussion on the limits and opportunities of participatory evaluation forms for sustainability in higher education.

2. State of the Art: Learning Competencies and Participatory Approaches

In this section, we begin by briefly introducing learning competencies for sustainability higher education programs to uncover how and in what way such competencies have been evaluated at the program level. We then briefly discuss the question of transdisciplinarity in the classroom, and finally introduce the notion of a participatory approach to evaluating learning programs.

2.1. Competencies for Sustainability: An Introduction

How sustainability higher education programs help deliver certain competencies is a growing area of research and action, bringing attention to a set of “specific and interrelated individual dispositions comprising knowledge, skills, motives, and attitudes” [12] (p. 5). A set of five key competencies provide a “reference scheme for transparently evaluating student learning and teaching effectiveness” [15] (see

Table 1. Key sustainability competencies.

Competency	Definition
Systems-thinking	A holistic approach to analysis that focuses on the way that a system’s parts interrelate and how systems function over time and within the context of larger systems. For example, graduates are able to “develop and test systemic interventions, transformational actions, and transition strategies toward sustainability, accounting for unintended consequences and cascading effects” [17] (p. 247).
Anticipatory	The ability to collectively analyze, evaluate, and craft rich ‘‘pictures’’ of the future related to sustainability issues and sustainability problem-solving frameworks. For example, graduates are able “to anticipate how sustainability problems might evolve or occur over time (scenarios), considering inertia, path dependencies, and triggering events; as well as create and craft sustainable and desirable future visions, considering evidence-supported alternative development pathways” [17] (p. 244).
Normative	This capacity is based on acquired normative knowledge, including concepts of justice, equity, social-ecological integrity, and ethics. For example, graduates are able to collectively map, specify, apply, reconcile, and negotiate sustainability values, principles, goals, and targets [15,17].
Strategic	The ability to collectively design and implement interventions, transitions, and transformative governance strategies toward sustainability. For example, graduates are able to develop plans that leverage assets, mobilize resources, and coordinate stakeholders to overcome systemic inertia, path dependencies, and other barriers to reaching envisioned outcomes [15,17].
Interpersonal	The ability to motivate, enable, and facilitate collaborative and participatory sustainability research and problem-solving. For example, graduates are able to “initiate, facilitate, and support different types of collaboration, including teamwork and stakeholder engagement, in sustainability efforts” [17] (p. 250).

). While the competencies are more general, they can be used in tandem with learning outcomes, the latter more detailed and specific, to help operationalize the competencies [17]. Authors recognize that key competencies for sustainability are important, but there is less consensus on what list of competencies to work with. Rieckmann [14] provides a selection of nineteen competencies, validated by academics in Europe, North America, and Latin America. De Haan [18] provides a list of eight competencies to enable active participation towards sustainable development, based on experiences in Germany. Several authors have built on the Wiek et al. [15] list to develop and revise their initial set of competencies: systems-thinking, values-thinking or normative skills, futures-thinking or anticipatory skills, strategic-thinking, interpersonal skills, and a sixth implicit competency which is the capacity of integrating the other competencies to solve sustainable issues.

A refining of key competencies was completed by Evans [7] through an extensive literature review followed by a case study based on a Bachelor of Arts in Sustainability Studies program at Colorado Mountain College. More recently, Brundiers et al. [12] explored the convergence of competencies for sustainability programs, employing the Delphi method to reach a final list of seven competencies, expanding the original five to include intrapersonal competency (based on the work of Giangrande et al. [9]) and implementation competencies [11]. Additional investigations were carried out to investigate ESD key competencies within different educational contexts, such as project managers [19], a framework as an assessment tool for professors [20], assessment of graduate sustainability learning outcomes [21,22], a literature review on the assessment of sustainability competencies [23], teacher’s perception on ESD competencies [24], a professional competency framework for ESD [20], and a systematic literature review on the state of the art research for ESD [25].

2.2. Applying Competencies to the Program Level

Evaluations related to specific courses within a given program are necessary for accountability to a faculty, broader institution, or the general public, involving quality control and continual improvement of the specific course [26]. Assessing whether the program as a whole is delivering on its objectives—in the case of the program under study, to form sustainability professionals—is also necessary. How to move from course-level evaluation to program-level evaluation is a growing field of inquiry and is largely rooted in the Scholarship of Teaching and Learning (SoTL). SoTL has long been concerned with improving teaching quality and student learning within specific contexts. Hubball, Pearson, and Clarke [27] find that much of the emphasis in SoTL literature has been on individual pedagogical practices and propose expanding to include experiences across a program as a whole, what they term the Scholarship of Curriculum Practice (SoCP). They argue that without SoCP, “programs become perfunctory or routinized, duplicative, or imitative […and] then these curriculum practices will cease to be effective” [27] (p. 51). While program evaluations are common, whether participatory or not, and generally address professional learning outcomes, a participatory utilization-based evaluation specific to sustainability competencies, proper to potential sustainability professionals, is less frequent.

O’Byrne et al. [8] conducted a review of 27 bachelor and 27 master sustainability programs, mainly in the United States and the United Kingdom, to assess how sustainability programs are structured, what courses and content are being taught in these programs, and the degree of similarity among the different programs concerning content and structure. While this study qualifies as a program-level assessment, there is no discussion around participatory methods for implementing the review. Giangrande et al. [9] developed an assessment process for SDG Goal 4 and Target 4.7. In the format of questionnaires, the evaluation framework was designed to be answered by educators. It was established on a set of key competencies and questions divided into five multiple intelligence categories: human rights, gender equality, the culture of peace and nonviolence, global citizenship, and cultural diversity. Evans [7] provides a roadmap of competencies and pedagogies for sustainability studies program development. The concepts developed were applied in a bachelor program at the Colorado Mountain College. Mapping of the program was elaborated, and the course descriptions were adapted in five competency diagrams serving as a framework for program development for sustainability education. Even though the mapping provided by Evans [7] is quite detailed, it serves more towards program development rather than evaluation. Participatory methods as an evaluation approach are not mentioned. Additional studies have investigated competencies at the program level in higher education, such as sustainability teaching, competencies, and pedagogical approaches [28], entrepreneurial education [29], and construction and project management professionals [30].

2.3. Transdisciplinarity and Participatory Approaches to Learning

Participatory approaches in higher education for sustainable development is a growing field of research and practice. Participatory approaches promote social learning, which is how individuals (students) come to understand their own and others’ values, and based on this changed understanding, act for the common good [31]. Encompassing a range of activities, participatory methods usually share an approach that seeks to engage different kinds of stakeholders in a dynamic process towards addressing a common question or problem. Such processes encompass interdisciplinary and transdisciplinary competencies [32]. Usually, they involve some form of collective deliberation on the problem at stake, the codesign of a proposed way forward, and an effort to address the problem in ways that are respectful of diverse opinions and ways of doing. Because participative methods engage people early on in a process and recognize contributions from diverse forms of knowledge, they can lead to more innovative outcomes and lasting results, reflecting the group’s commitment. That being said, participatory methods can take different forms in terms of the level of involvement of stakeholder groups and are perhaps best understood on a continuum [31]. Participants can be involved as sources of information, data, feedback, or as significant contributors to the research involved throughout the design phase. Thus the degree of transdisciplinarity can range from consultancy to full participation or participatory-transdisciplinarity [33]. We assume that a higher degree of involvement of participants could lead to enhanced social learning.

Brundiers, Wiek, and Redman [34] investigated real-world learning opportunities in terms of ESD competencies. Among the opportunities mentioned by the authors, the transacademic research educational approaches, such as transdisciplinary, participatory, and community-based approaches performed by the students and professors in partnership with nonacademic experts, were seen as valuable educational experiences for students. Balsiger [35] describes a transdisciplinary research project explored by students in a classroom setting, commenting on how to move from soft transdisciplinarity—based on shallow integration and narrow collaboration—and moving towards inclusive transdisciplinarity by increasing the number of stakeholders, for example. Caniglia et al. [36] propose an experience-based, student-centered learning framework for early-stage (novice) sustainability competency development in higher education programs that highlights the value of experiential learning and bottom-up approaches in developing competencies for sustainable development. Still, their research does not report on participatory evaluations at the program level. Sipos, Battisti, and Grimm [37], developed a matrix for program evaluation suggesting that learning objectives, such as transdisciplinary curriculum and a democratic and participatory learning environment, can be integrated across existing curricula for an effective transition to a sustainability-oriented higher education or transformative sustainability learning. The authors do not mention if the students could perform the evaluation.

2.4. The Participatory Evaluation of Learning Programs

A cornerstone of participatory methods (in evaluation or otherwise) is, as the name suggests, the participation in the design and execution of the subjects being studied [31]. As such, participatory methods seek not only to obtain “richer” answers but also to empower “nonexperts” through social learning [31]. Hence, in our understanding and specific context, the participatory evaluation seeks to allow the least influential stakeholder (our students) in a given program to codesign the evaluation approach and parameters [38]. This evaluation form has been subcategorized as empowerment evaluation [39], as it seeks to codevelop a solution to commonly identified issues in a given program. While participatory program evaluations are not a novelty, assessing a sustainability teaching program against a specific set of competencies for aspiring sustainability professionals is less common.

Holm et al. [40] discuss a participatory process towards understanding how a quality assessment method can be applied to ESD. A group of practitioners from diverse Nordic universities came together to develop a process, which included a “check” or assessment phase. There is no discussion on whether the assessment phase could be carried out by engaging students.

Bergsmann et al. [10] reflect on participatory approaches in competence-centered curricula evaluation. The authors highlight that current evaluation methods focus on single competencies or specific aspects of the curriculum and often fail to reflect on stakeholder needs, which they find suboptimal in the context of the increasing tendency to design curricula as learner-centered, particularly in education for sustainable development. They suggest (1) defining the ideal competencies a student should acquire through the program, (2a) ensuring that the teaching process allows for the development of these competencies, (2b) ensuring that said competencies are actually being developed by the students, and (3) analyzing and intervening at any level where (2a) and (2b) might not match while including all relevant stakeholders in these stages, through focus groups and online surveys. Bergsmann et al. [10] propose a set of strategies that may aid in overcoming institutional resistance and suggest that participatory evaluation in learner-centered programs may assist in informing institutional development processes, that is, ease and improve competency-based higher education curricula. The authors thus propose a method for participatory evaluation at the programmatic level, which we build on in our study. Further, Lozano et al. [41] suggested a framework to connect pedagogical approaches and competencies in an extensive literature review. In their review, participatory research action is highlighted as a pedagogical approach, similar to action learning, that contributes to the student’s capacity to address social justice and community-building issues. These insights were applied to a participatory program evaluation.

3. Methods: The Participatory Evaluation of a Master Program

In the evaluation of a Master program on sustainability at the University of Geneva, we set out to assess in what way the learning outcomes at a program level could be related to the five main competencies of Wiek et al. [15]. We do not assume that students come to the program with a blank slate of competencies, but we consider how the program either introduces or enhances these competencies among students. To do so, the professor responsible for a class on “Policy, Program and Project Evaluation” asked his students if they would be interested in carrying out the assessment of the overall program as part of their learning experience in his class that semester (Spring 2020). By engaging in an evaluation of their own program, the following main question was agreed upon together: what competencies for sustainability does the program deliver? An additional question also emerged: what do the students gain from engaging in the evaluation themselves? Through the process, we reflect on the relevance of the implementation competency suggested by Brundiers et al. [12], relating to a hands-on approach to implementing strategic actions through learning by doing.

The Master of Arts (MA) in Sustainability is a 120 ECTS sustainability teaching program offered at the University of Geneva (UNIGE) in the social sciences faculty (SdS) and Sociology department. The Master program was initially designed in partnership between UNIGE and the International Organization for Standardization (ISO) and launched in 2011. It is intended as a program intersecting between academia and the professional world, aiming to mainstream sustainability in organizational processes and train future sustainability professionals and change managers. The program is interdisciplinary, and its learning outcomes are articulated around three main pillars: standardization, social regulation, and sustainable development. It offers a conceptualization of sustainability beyond a solely environmental dimension to include economic, political, and social dimensions, all of which are needed to understand how action towards the normative aim of sustainable development can be coordinated across different sectors. The program is designed to deliver 120 ECTS credits, being 96 compulsory credits (including the Master thesis or internship report, composed of 24 ECTS credits) and 24 ECTS credits from subjects to be chosen by the students according to their areas of interest. In summary, the wide range of subjects addresses critical global issues such as global health, reforming corporate behavior and practice, the global financial and economic crisis, environmental policy, and issues linked to governance and public accountability. Further information about the program, such as detailed course descriptions, schedules, regulations, study plans, and intended learning outcomes, are publicly available on UNIGE’s website [42]. Due to its high level of interdisciplinarity, the Master program welcomes students from various academic backgrounds (though primarily social scientists) and from all over the world.

The program has changed throughout its existence, due to on the one hand professors’ experiences, turnover in teaching staff and course offerings, as well as the introduction of new programs addressing similar issues, and continual improvement efforts. While changes in longstanding programs are inevitable, these may have altered the actual sustainability competency outcomes originally intended by the program designers. In order to capture what professors intend with their courses, the inquiry was centered on which sustainability competencies their courses deliver (discussed further in

Table 2. Inquiry results related to five key sustainability competencies: students and teachers.

Stakeholder	Scale	Systems-Thinking	Anticipatory	Normative	Strategic	Interpersonal
Students (Question: The Master program has helped you develop…) (n = 27)	Strongly Agree	26%	6%	32%	12%	23%
	Agree	56%	68%	62%	47%	62%
	Disagree	15%	23%	6%	41%	15%
	Strongly Disagree	3%	3%	0%	0%	0%
Teaching staff (Question: do you deliver …through your course?) (n = 11)	Included in all my lectures	18%	0%	18%	18%	0%
	Included in more than half of my lectures	46%	55%	55%	36%	18%
	Included in less than half of my lectures	27%	18%	18%	36%	55%
	Not at all included	9%	27%	9%	9%	27%

). Such a self-assessment would benefit from being complemented by in-depth evaluations of each course in terms of learning outcomes, and how they relate to the development of specific sustainability competencies. While such an evaluation is out of the scope of this study, it is currently being undertaken for the program—as one of the results of the preliminary evaluation. While courses in the program are evaluated each semester, there has not been a programmatic-level evaluation in several years, and none of these evaluations considered competencies for sustainability. In the past, program-level evaluations have been handled by the pedagogical support center at the university; for this exercise, members of this center were consulted on the proposed evaluation design.

First, the professor responsible for a class on Policy, Program and Project Evaluation proposed a participatory assessment of the program by its students to the scientific committee of the Master program. This evaluation aimed to review the program, its contents, and its resources, to inform potential future direction of the program—in a context where more and more programs on sustainability are emerging, in Switzerland and elsewhere. Once this was approved, the students themselves were asked to agree to such an approach; critically, they also had to agree to work in several groups on different aspects of the assessment and receive one final and common grade for the class. There was thus an emphasis on collaboration and collective work rather than competition. The class first discussed the different learning competencies for sustainability in higher education and selected the Wiek et al. [15] list. The class then identified vital stakeholders through a joint brainstorming session. Five broad categories were identified: students, alumni, teaching staff (professors, assistants, external lecturers), university management (including the rectorate, dean’s office, the program’s steering committee, and UNIGE’s communication department), and possible employers of alumni. A total of 113 people were consulted. A mapping exercise was also conducted to identify programs with similar learning offers and compare this program to them in terms of their overall orientation (scientific, economic, social), their attractiveness (ratio of applications to enrolment), and their geographical location. (The mapping exercise is beyond the scope of this paper and not detailed here, as no direct link was made to competencies).

The program evaluation involved a mixed-methods approach, including quantitative inquiries, semistructured interviews, and desktop research, summarized in Appendix A,

Table A1. Summary of Data Collection Methods and Response Rates (n = 113).

Stakeholder Group	Data Collection Method(s)	Survey Description	Response Rate	Comments
Mapping	Desk research and admissions ratio collected by email		28 programs identified
Alumni	Quantitative/survey	Thirty-five questions were divided into 4 sections: the general appraisal of the program, professional satisfaction, recommendations for the program, and sociodemographic questions. Five questions in the general appraisal section are based on the competencies developed by Wiek et al. [15]. Responses based on a three- or five-point Likert scale, multiple-choice questions, and space for comments after each question. Example: While doing the master, did you feel connected and genuinely motivated by the master program curriculum?	20 responses	94 alumni in total, 46 contacted
Students	Quantitative/survey	Thirty questions in total were divided into 2 sections (academic content and sociodemographic questions). Five questions in the educational content section are based on those key competencies developed by Wiek et al. [15]. Responses based on a three- or five-point Likert scale, multiple-choice questions, and space for comments after each question. Example: The program has helped you develop “anticipatory competencies”, meaning the ability to collectively analyze and picture future scenarios related to sustainability issues? (Likert scale)	52 responses	56 students in total, first and second year
	Qualitative/focus group (second phase)	Discussion on (1) the strengths and weaknesses in the participatory and student-led process of evaluating the course program and (2) what skills students were able to develop through this course evaluation. Example: Did you find the participatory evaluation process useful, and if so, what have you learned? In terms of the evaluation itself, do you feel that this was a success or a failure? Participants’ self-assessment of sustainability competency development (Likert scale).	8 participants	27 students contacted to participate in the evaluation
Teaching Staff	Quantitative/survey	Thirty-eight questions were divided into 5 sections: general thoughts on the program, inclusion of key sustainability [15] competencies, additional questions for guest speakers, additional questions for visiting professors, and personal data (teaching experience, title, department, etc.). Example: To what degree do you believe that your course addresses themes related to sustainability (e.g., climate change, poverty, global health, etc.)? (Likert scale)	11 responses	16 teachers contacted
	Formal class evaluation survey	Summative course evaluation tool adopted by the rectorate as a part of the university’s quality control policy	7 forms
Management	Semistructured interviews	This survey had three main areas: orientation, curriculum, and admissions and communications (enrollment procedures, communication tools). Each interviewee was provided minutes of the interview for comments, corrections, and approval, to ensure transparency and avoid any issues related to data protection. Example: Which of the “key competencies of sustainability” in your opinion have been developed in this program?	11 responses, 9 interviews	20 members of management contacted
Employers	Semistructured interviews	Focused on the skills and knowledge needed to succeed in a sector, department, or organization, employers’ opinions on program graduates, and expectations on skills and knowledge for the future. Example: What are the skills needed to succeed in your sector?	12 responses	20 employers contacted, with permission of alumni

. While the intention at the planning stage was to complement the quantitative inquiries and qualitative interviews with focus groups, this proved impossible due to the semiconfinement measures taken during the COVID-19 pandemic in Switzerland. The evaluation team adapted to the measures in place by soliciting virtual interviews or written feedback. The evaluation drew upon the key sustainability competencies as defined by Wiek et al. [15,17], as this set of competencies is specific to the field of sustainability and would result in a distinct and recognizable set of qualifications for sustainability graduates. Students worked in groups with different audiences and delivered reports for their tasks, which were then discussed and brought together in a final report drafted by the teaching assistant and in collaboration with the program instructor and program director. Students received the final report and were invited to comment. They also presented their findings to the steering committee of the Master program. In the last phase, several months after the finalization of the report, a focus group was conducted with students to understand how this participatory form of program evaluation delivered certain competencies for sustainability. This last phase allowed us to reflect on the process of student-led, practice-based assessments. At the time of writing, the results are being integrated in a program overhaul, which also involves an in-depth review of all course offerings.

4. Research Results

The research results are organized around two main themes: In part 1, we discuss what learning competencies are delivered by the Master program based on the student-led assessment. In part 2, we uncover in what way the assessment process also served to enhance certain competencies.

4.1. Competencies Delivered by the Program: A Student-Led Evaluation

The Master program was evaluated against different objectives, including how it compares to other similar offers (the mapping exercise). Students as a stakeholder group were consulted to understand why they were attracted to the program (background and geographic place of origin) and their overall satisfaction with the program offer. Concerning the five competencies detailed in Table 1, students were asked whether they (dis)agreed that the program had helped them develop these competencies, while teachers were asked whether they felt their classes deliver these competencies through their lectures. Understood as being more than a sum of its parts, a list of competencies is most useful when competencies can be combined in meaningful and effective ways [15]. The results are summarized in Table 2. Here, it is essential to note that two different scales of analysis are combined: students are responding to the program as a whole, while teaching staff are replying for the courses for which they are responsible.

Teachers in the Master program were also asked a series of questions, such as the distinctiveness of the Master program compared to other similar programs or areas for improvement at the program level. For their classes, they were asked how their lectures and teaching approaches contribute to delivering an interdisciplinary and multidisciplinary approach to sustainability. They were then asked a series of questions related to the Wiek et al. [15] list of five competencies. Generally, teachers claimed that the competencies were addressed in their teaching offer. Perhaps the one discrepancy relates to the strategic competency or the ability to collectively design and implement interventions, transitions, and transformative governance strategies toward sustainability. Students were less likely to claim that this competency was delivered through the program.

The management stakeholder group was asked about the key sustainability competencies through interview questions. This group views systems-thinking and interpersonal competencies as the most developed through the program’s interdisciplinary nature, the number of group projects, and collaborations with external organizations, followed by normative and strategic competencies, and finally anticipatory competencies.

Despite these positive results, it was observed that the definition of some competencies was not entirely understood by all participants—even if each competency was provided through a detailed description in the questionnaire. This is proven by specific comments left by the students, such as viewing systems-thinking solely as a macroeconomic issue or strategic competencies as being predicated on statistical analysis. This effect was expected to some extent as, even in the academic literature, there is a lack of consensus on the meaning of sustainability and related concepts. For example, White [43] found over 100 definitions of sustainability in the literature. A lesson learned is that professors should engage in promoting a shared vision on the meaning of sustainable competencies. Finally, members of the management group stressed the need for the ability to work with people from various fields and backgrounds without making a link to the five competencies—specifically, strategic and interpersonal skills.

Generalized and abstract competencies accommodate the topical knowledge necessary for problem-solving in a specific context, while learning outcomes, specific and detailed, offer a way to operationalize these competencies [17]. Since this was the first time the Master program was assessed against competencies for sustainability, it is difficult to relate the program’s learning outcomes in its current form to the specific learning outcomes of the evaluation. This is one of the improvement points identified, the need to perform the assessment of competencies and respective learning outcomes on a regular basis in parallel with the equivalent related to sustainability.

Sustainability teaching programs aim to arm graduates with a set of competencies that will contribute to solving some of humanity’s most pressing issues and aid them in finding gainful employment. According to the employers interviewed, the Master program provides a good notion of what sustainability is, but the program’s graduates tend to lack managerial and operational skills, such as content writing, adaptability, conflict management, and the ability to work with diverse stakeholders. This response is corroborated by the comments left by teaching staff, who would like to see a variety of perspectives integrated on sustainability issues. Finally, all stakeholder groups emphasized the need to alter the didactical approach to increase “learning by doing”. This underscores the relevance of the implementation competency suggested by Brundiers et al. [12], relating to a hands-on approach to implementing strategic actions through learning by doing. This participatory approach certainly falls into this category; as such, we now turn to how and in what way the student-led evaluation process contributed to different competencies for sustainability in higher education.

4.2. Competencies Delivered through the Student-Led, Practice-Based Evaluation

A follow-up discussion group took place several months after the end of the evaluation process, once the final report had been compiled by the teaching assistant, with the input of two professors involved in the Master program. The aim of the discussion group was twofold: to assess (1) the strengths and weaknesses in the participatory and student-led process of evaluating the program and (2) what skills students were able to develop through this evaluation.

For most students, this exercise was their first formal evaluation of any kind, let alone a participatory evaluation of their own Master program. Overall, they appreciated the opportunity to participate in a program evaluation and the fact that this evaluation will serve as a basis for strengthening the program in the years to come. An essential aspect that several participants highlighted is the notion of empowerment: several participants felt empowered by the opportunity to talk about the weaknesses of their program with the scientific committee and the dean of the faculty. Students felt that they were able to provide constructive criticism of certain aspects of the program. This was seen as a critical strength of the participatory evaluation process. When asked whether they felt that the evaluation itself was a success or a failure, several participants stated that the answer to this question would depend on whether changes along the lines of the recommendations will be implemented. Indeed, according to Bergsmann et al. [10] (p. 7), “the aim of [participatory evaluations] is the sustainable improvement of competence-based teaching in higher education institutions. Hence, the evaluation results should be used in decision-making”. This is a critical issue: in participatory methods, when people are engaged in resolving a common problem, there are raised expectations around addressing the problem through these commonly agreed-upon solutions or recommendations. We are glad to report that many of the recommendations highlighted by the students have been taken into account in the reorientation of this Master program.

In terms of what skills students were able to gain through this process, students spontaneously mentioned the development of communication skills and new competencies developed concerning specific software used for developing and analyzing survey data. These can be interpreted as strategic and interpersonal skills. However, some students felt they were lacking a better understanding of the overall process involved in conducting an evaluation. Indeed, the class was divided into groups corresponding to stakeholder categories identified collectively at the start of the course in mid-February. While one participant found it positive to focus on specific aspects of the evaluation, the division of the class into groups left the majority feeling disconnected from one another, making it difficult to see the big picture. As such, they may have lacked the opportunity to develop systems thinking or the interrelation of the different stakeholders under evaluation. One of the reasons why the groups may have felt disconnected from one another had to do with the extraordinary circumstances of the spring 2020 semester: as of mid-March, the University of Geneva switched to online and virtual teaching for the rest of the semester (ending in June). Thus, students had less opportunity to interact together in the classroom informally and were forced to work primarily within their different groups. Perhaps because of this issue, some students felt that they would not be able to conduct a program evaluation independently.

That being said, the students tended to downplay their learnings from the participatory evaluation process. They claimed not to have learned much; and yet, they learned to self-organize online, coordinate outreach to organizations and individuals for interviews, and design, develop, conduct, and analyze interviews and surveys. More generally, they learned to work together and practice different theories and approaches on program evaluations. They also developed final reports and managed to do all of this during a pandemic, with semiconfinement measures in place, and over the course of five months. One participant stated feeling positive about the evaluation itself, given that they started from scratch in February and were able to present their results to the committee in June. As can sometimes be the case in experiential or learning-by-doing approaches, participants can be frustrated and feel that they may not have learned anything new when, in fact, they developed new skills and competencies without realizing it [44]. On a final note, there were also different levels of engagement in the participatory process. Some students did not feel as personally involved in the process, ultimately viewing it as an academic course for credit. While for others, there were expectations that more would come out of the evaluation—over the summer months, for example—and that there was insufficient communication on the status of the evaluation process and next steps.

The participants were asked to assess whether this evaluation had aided them in improving on those competencies identified by Wiek et al. [15] as key sustainability competencies. A poll was conducted, based on a Likert scale of 1 to 5 (1 being the lowest and 5 being the highest level of competency development). Unsurprisingly, the participants felt that the competence they had most improved upon was their interpersonal competence (mean: 3.38), as was gleaned from earlier statements, followed by strategic thinking (mean: 2.50). Normative thinking was found to be the competence the participants felt they had the least developed (mean: 1.63), followed by systems- and futures-thinking (mean: 1.75 respectively).

5. Conclusions and Discussion

UNESCO’s [1] definition of ESD highlights the notions of lifelong learning and empowerment. Indeed, students are learners, consumers, and citizens, and thus potential activists, able to mobilize political action to advocate for a cleaner environment and fairer societies [5]. ESD is a growing field of interest, be it academically or politically, and there is significant literature focusing on the evaluation of courses in terms of the competencies they deliver (e.g., [15]). Still, few studies [7] focus on how to assess whether teaching programs on sustainability achieve their aims. The evaluation of the Master program by its own students in the context of a semester-long course on Project, Program and Policy Evaluation aims to contribute to this literature gap while also serving as a manner to reflect on the benefits of participative evaluation methods. Students were invited to evaluate their Master program as a real-world learning experience.

The limits to the research design lie in the fact that competencies were uncovered by asking different actors a series of questions, where each of the five competencies was explained in detail. While students, teaching staff, management, and future employers were given the time to reflect on the competencies, the assessment did not involve studying course curriculum to assess how and in what way competencies were being delivered in practice; therefore, a suggestion for further studies is to perform this participatory evaluation for each course and contrast the sustainability competencies identified with the assessment of the whole program; observations in the classroom were also lacking, but would have been found to be rather intrusive. Another challenge related to the study of competencies at the program level is that the whole is often more than a sum of its parts: while we can evaluate competencies given by different courses in a program, it is much less evident to assume that students simply take away these added competencies. Being part of a program also involves extracurricular activities, such as socializing and debating with colleagues and teachers outside of the classroom setting, or through internships and work opportunities—or what are considered to be informal learning competencies, following Barth et al. [45]. How to capture the competencies gleaned from such social interactions is less obvious.

Competencies must also be understood as more effective when they are combined. The evaluation of the student-led assessment is thus an interesting case in point, as several competencies were brought together. While not part of our original list of five competencies (building on Wiek et al. [15]), the importance of implementation skills as a sixth competency, as proposed by Brundiers et al. [12], was underscored through the participatory evaluation process. Participating also means setting expectations about how findings will be utilized. By engaging students in such a process, the students take on the dual role of being both the assessors but also the main actors who have much to gain and lose from any changes to the program. Thus, the student involvement must extend beyond the assessment phase—with some form of student representation ensured in the different decisional instances that might be planned for regarding changes to the program in the future. As the program prepares for a reorientation in 2022, students are represented in the scientific committee that will steer this process. There is also the commitment to ensuring that competencies for sustainability are reflected in the design of the new program’s course offering towards delivering a program that will prepare students for tomorrow’s sustainability challenges and opportunities.