**Identifying and Overcoming Barriers to Integrating Sustainability across the Curriculum at a Teaching-Oriented University**

**Brian Pompeii 1,\*, Yi-Wen Chiu 2,\* , Dawn Neill 3, David Braun 4, Gregg Fiegel 5, Rebekah Oulton 5, Joseph Ragsdale <sup>6</sup> and Kylee Singh <sup>7</sup>**


Received: 17 March 2019; Accepted: 30 April 2019; Published: 9 May 2019

**Abstract:** This research collects and analyzes student and faculty knowledge and perceptions toward sustainability education at a predominately undergraduate, teaching-oriented university. In-depth, qualitative methods distinguish low- and high-knowledge student and faculty cohorts, identify perceived barriers to sustainability education in each cohort, and recognize strategies to overcome the barriers identified by each cohort. Data collected from recorded and transcribed semi-structured interviews of student and faculty subjects underwent analysis via repeated readings to uncover key themes. Results required developing metrics for student and faculty sustainability knowledge and attitudes across disciplines, determining discipline-specific gaps in sustainability knowledge and differences in attitudes, and relating implementation barriers to general or specific knowledge gaps and attitudes. Findings identified low and high levels of sustainability knowledge within the student and faculty subject population and revealed barriers in pursuing interdisciplinary sustainability curricula across disciplines and among both students and faculty at the study university. Overall, higher sustainability knowledge participants tend to identify barriers related to institutional accountability while lower sustainability knowledge participants tend to identify barriers related to personal responsibility. Distributing barriers and solutions along a continuum from personal responsibility to educational institution responsibility reveals more recognition of barriers at the personal level and more solutions proposed at the institutional level. This result may reflect a common tendency to deny personal responsibility when addressing sustainability challenges.

**Keywords:** sustainability education; qualitative research; interviews; implementation barriers

#### **1. Introduction**

#### *1.1. Context and Background*

This research contributes to the broadening understanding of impediments to integrating sustainability education into higher education. Prior studies have investigated structural conditions ranging from educational priorities to disciplinary silos to competing values [1,2]. This study seeks to understand the relationship between a level of sustainability knowledge and perceived barriers to integrating sustainability-based instruction in higher education. Specifically, this project explores the perceptions of students and faculty regarding issues of sustainability education and identifies potential barriers to implementing the teaching and learning of sustainability at the university. As such, this study identifies barriers and solutions to the implementation of sustainability among different sustainability knowledge groups of faculty and students.

The focus institution is California Polytechnic State University, San Luis Obispo (herein referred to as Cal Poly). As a non-PhD granting and predominantly undergraduate university, Cal Poly enrolls approximately 22,000 students in six colleges with an emphasis on hands-on pedagogy to prepare students for the job market and "success in a global economy" [3]. Work aimed at advancing sustainability education and curricula at the university accelerated with the university's signing of the Talloires Declaration in 2004 [4,5]. The resulting action plan committed Cal Poly to "sustainability and environmental literacy in teaching, theory, and practice". The university took steps to advance this plan with the establishment of the Sustainability Learning Objectives (SLOs). The SLOs promote the idea that all graduating students should have some knowledge of fundamental sustainability principles. The Academic Senate Resolution 688-09 establishing the SLOs states [6]:

"Cal Poly defines sustainability as the ability of the natural and social systems to survive and thrive together to meet current and future needs. In order to consider sustainability when making reasoned decisions, all graduating students should be able to:


In 2014, the California State University (CSU) sought to further advance sustainability education for all its campuses (including Cal Poly) when it updated its sustainability policy [7]. The policy states that the "CSU will seek to further integrate sustainability into the academic curriculum working within the normal campus consultative process." Cal Poly more recently signed the Second Nature Climate Commitment, stating that "Cal Poly is committed to achieving carbon neutrality and climate resilience as soon as possible, and is infusing this work into curriculum, research, and student experience."

To support the advancement of sustainability education on campus, the Center for Teaching, Learning, and Technology at Cal Poly formed an interdisciplinary faculty learning community in 2016 focused on "Teaching Sustainability Across the Curriculum." This faculty group, representing four of six academic colleges, works to improve students' sustainability learning through the creation and promotion of educational experiences based on current best practices. Within the group discussions, anecdotal evidence and faculty experiences pointed to a consensus that implementation of sustainability goals was at best limited in the current campus climate, despite ongoing institutional efforts. Therefore, a campus-wide survey was proposed to assess student and faculty sustainability knowledge and awareness in order to make more informed future decisions.

Concurrent with the development of the survey, Cal Poly applied for certification through AASHE/STARS (Association for the Advancement of Sustainability in Higher Education/Sustainability Tracking, Assessment and Rating System) receiving a silver rating (62.57 of 100 possible points) in February 2017. This rating considers six domains of university sustainability: Institutional

characteristics, curriculum and research, engagement, operations, planning and administration, and innovation and leadership. Cal Poly received only 28.13 of 40 possible points in the curriculum section, with two notable curricular areas contributing to this result—the lack of sustainability-focused and -related academic courses available (6.13 of 14 points) and the absence of assessment of sustainability literacy (0 of 4 points). The results indicate that only 4.9% of courses at Cal Poly are considered sustainability course offerings. Zero points were scored in the category of sustainability literacy assessment, because, at the time of submission, an annual assessment of students' sustainability knowledge did not exist. These scores reveal that while Cal Poly has theoretically dedicated itself to sustainability education, it is unclear how related policies and commitments materialize within the curriculum.

This study seeks to understand how the perception of barriers to and solutions for the integration of sustainability in teaching and learning correlates with sustainability knowledge, in order to identify opportunities for improving sustainability education. To achieve this goal, students and faculty from across the six colleges were assessed using qualitative methods to determine in-depth understanding of both sustainability knowledge and the identification and overcoming of barriers to integrating sustainability in higher education curriculum.

#### *1.2. Literature Review*

Multiple studies reported in sustainability education literature contribute to the integration of sustainability in the curriculum [8–10]. Although the need to assess sustainability across campus has been emphasized [11–13], former studies fall short either at pointing to a precise method of assessment or taking into account the context of sustainability knowledge. The literature does, however, reveal that sustainability learning outcomes can vary greatly even within environmental based courses and suggest further research on disciplines and majors that have historically been on the periphery of sustainability education [14]. An immense survey-based, quantitative study in European higher-education institutions also investigated the relationship between different pedagogical approaches and learning outcomes or competences. Results found that none of the competences examined were likely to address sustainability in any three of its dimensions (economic, social, or environmental) [15].

The literature identifies barriers internal to universities that prevent infusing sustainability: Financial constraints, lack of understanding and awareness of sustainability, resistance to change, and difficulty achieving a "coherent institutional approach, where operations, teaching, research, and outreach are synergized" [16]. The literature contains several examples of how silos in academia tend to act against infusing sustainability. According to Miller et al. [17], academic institutions typically organized around scholarly disciplines lack the "epistemological pluralism and reflexivity" required producing sustainability knowledge characterized by "social robustness, recognition of system complexity and uncertainty, acknowledgement of multiple ways of knowing and the incorporation of normative and ethical premises." Others also state that academic silos represent the most insidious barrier, because specialization helps to isolate faculty and "prevents the systems-level integration required to embed sustainability" [16].

Beyond silo-ing, other institutional level barriers have been identified, including institutional priorities and external pressures [18]. For example, perceptual barriers include the competition for funds on campus, the commodification of education, and the exclusion from any faculty evaluation criteria [2]. Institutional barriers to the comprehensive adoption of sustainability in higher education curriculum also include differences in understanding of the concept of sustainability and challenges of working across all areas of university structure [19]. An evaluation of faculty participation in the University of Vermont's Sustainability Faculty Fellows program examined the impact of a funded faculty learning community focused on enhancing sustainability curricula across disciplines [20]. Results identified the largest barriers for faculty included: A packed curriculum, lack of planning

time, lack of department support, difficult to integrate into content, lack of content knowledge, lack of learning activity resources, and class size [20].

Arizona State University's School of Sustainability provides an example of an approach where an institution successfully applied an adaptive cycle to create a sustainability program emphasizing "interdisciplinary collaboration and community engagement" [17]. The literature offers several approaches to distinguish individual from institutional responsibilities towards infusing sustainability. A proposed sustainability compass depicts five axes of individual and institutional elements required to foster sustainability knowledge [17]. Similarly, Sterling's model for integrating sustainability in education distinguishes "bolting-on" by adding separate sustainability courses from the deeper level of integration via "building-in", which educates for sustainability by teaching sustainability issues in discipline-specific courses [21,22].

Our research is built on broad based projects like Lozano et al. [15] with an in-depth textured analysis of student and faculty experiences, in order to examine a level of sustainability knowledge in relation to the identification of barriers and solutions to further integrate sustainability into the curriculum. This approach involves categorizing interview participants' responses based on their level of knowledge in sustainability.

#### **2. Materials and Methods**

Given the lack of existing data on sustainability knowledge among Cal Poly students and faculty, qualitative methods were deemed the most appropriate for data collection and analysis. Data were collected using semistructured interviews [23], in which a set of open-ended questions were prepared to guide the interview process but might be asked in a particular order or format. Interview questions were designed to gauge each participant's general sustainability knowledge and behaviors, to assess how sustainability is approached as a learning objective across disciplines, and to identify potential barriers to teaching sustainability across the curriculum. A total of 17 faculty and 39 student interviewees from six colleges at Cal Poly (i.e., agriculture, architecture, business, engineering, liberal arts, and science and math) voluntarily participated in this survey. Students were recruited from large general education (GE) courses within a variety of disciplines and provided minimal assignment extra credit incentives for participation. The large GE courses chosen were defined as courses with over 125 students where all academic departments were represented in the possible student pool. Recruitment announcements were made in four such classes. Third-year and fourth-year students were specifically targeted as they would have more class experience to draw upon.

There are several qualitative data collection practices for conducting interviews based on what type of data the researcher wants to collect [24,25]. This project used a purposeful interview sampling technique, which has been recognized as a powerful tool to capture empirical relationships between different groups of the data [26]. In qualitative research, sample size has been shown to be less important when the participants have personal experience with the project subject, when small numbers of participants are studied intensively, and when the type of participants are chosen purposefully [27]. Moreover, this is not a hypothesis-based study, and the selected method does not aim for deriving statistical significance to test any predeveloped hypothesis. The responses from the semistructured interviews provided considerable data for analysis, including over 10 h of recorded transcripts, which serves the purpose of the study despite the small sample size for both students and faculty.

#### *2.1. Interview Design and Implementation*

Interviews were conducted by a small team of student researchers. Prior to commencing data collection, all student researchers participated in an in-depth training session with faculty researchers to ensure interviewer consistency. The same faculty researchers were present during all interviews to further ensure consistency and maintain rigorous oversight of data collection. Each interview took approximately 10–20 min to complete. All interviews were audio-recorded and transcribed. The transcripts were individually coded for emergent themes using a grounded theory approach [28]. This approach allows the researchers to determine patterns on how interviewees perceive sustainability in academia. Coding and analysis relied primarily on assessment by three faculty researchers with experience in qualitative methods to ensure inter-rater reliability. The semistructured interviews were designed to assess each participant's knowledge of, perceived importance of, and exposure to sustainability concepts and practices, with the following questions guiding that conversation:


#### *2.2. Transcript Analysis*

The stage of analysis in this study was conducted by utilizing several established techniques. Ryan and Bernard (2003) list several techniques for identifying themes when analyzing qualitative data [29]. Interview transcripts were analyzed for the following themes: Repetitions, indigenous typologies or categories, similarities and differences, missing data, and theory-related material. Recognizing repetitions is one of the most commonly used procedures for identifying themes in interviews [30–32]. Multiple, collaborative readings of the transcripts allowed for the identification and marking of statements that succinctly characterized the repeated themes.

Data analysis relied on coding, an iterative methodology identifying text "that captures and signals what is going on in a piece of data in a way that links it to some more general analysis issue" [33]. Coding schemes provided a framework for identifying emergent themes linking specific data points to the broader concepts under investigation. Following the development of a coding scheme, analyses were then incorporated to identify emergent themes, derive explanations, and actionable responses related to main research objectives [33,34]. In this study, data analysis was conducted by multiple researchers in order to avoid interpretive bias from a single researcher in the coding process, thereby gauging inter-rater reliability and establishing qualitative rigor [35,36].

Transcription analysis consisted of three phased readings. The entire interdisciplinary research team carried out an initial reading to develop a tentative, emergent coding scheme based on the repetition of certain ideas. A second reading was carried out with a smaller group of three researchers, each with expertise in qualitative methodologies. During the second reading, each researcher first coded each transcript for level of sustainability knowledge. These researchers then engaged in group discussions that gauged and normalized transcripts for either high or low sustainability knowledge. The same three qualitative researchers then completed a third reading, individually coding the text according to the coding scheme developed by the entire research team, then analyzing codes for emergent themes related to barriers or solutions. The researchers then engaged in group discussions to reach consensus on the key actionable emergent themes. Data saturation was achieved, indicating that further interviews would have produced similar results [37].

#### **3. Results**

All participant responses were reviewed and analyzed for determining high or low level of sustainability knowledge through analysis of the introductory questions "How do you gauge your own knowledge of sustainability" and "How do you define sustainability?" A high or low level of sustainability knowledge was determined through phased readings and defined through

researcher congruence. Researchers referenced common definitions of sustainability including: Cal Poly's definition of sustainability "the ability of the natural and social systems to survive and thrive together to meet current and future needs", the Brundtland Commission's statement on sustainable development "meeting the needs of the present without compromising the ability of future generations to meet their needs", and references to the 'three Es—Environment, Equity, Economy'. High knowledge had a relatively low threshold for connection with agreed-upon definitions. Any mention of a broad understanding of sustainability was rated as high. When identified according to the structure of observed learning outcomes (SOLO) taxonomy, high knowledge responses contain multistructural, relational, or extended abstract statements, whereas low knowledge responses operate at the prestructural or unistructural levels [38]. Thus, participants responding with general or greater information implying broader or more comprehensive perception to the question "How do you define sustainability?" were defined as "high". Responses indicating a high level of sustainability knowledge included:

*"Meeting the needs of the present without compromising the ability to meet the needs of the future"*

*"* ... *it's the practice or philosophy that resources should not be used up so that any kind of practice or any materials that are used, should be used in such a way that the resource doesn't get depleted for the foreseeable future or for infinity."*

*"Sustainability has to do with making sure that the way that humans live, the resources we use* ... *the inputs and outputs of our society are things that could continue for thousands of years without a problem."*

Responses demonstrating a low level of sustainability knowledge were those that did not recognize a larger philosophy or were simply unrelated to the question asked. For example, if responses simply eluded to activities such as recycling or driving hybrid cars, these would be classified as low knowledge. Of the 39 student responses, 22 were noted as having a high level of sustainability knowledge, and 17 were noted as having a low level of sustainability knowledge. Of the 17 faculty responses, 10 were noted as having a high level of sustainability knowledge, and 7 were noted as having a low level of sustainability knowledge.

Participants were grouped in this way in order to develop a deeper understanding of how their prior interest and/or knowledge regarding sustainability might impact identification of issues associated with sustainability in the Cal Poly curriculum. An a priori assumption was that sustainability 'adherents' (i.e., those students and faculty with prior or continued exposure to sustainability education) would represent a qualitatively different subset of responses with a generally more positive attitude toward sustainability education due to their understanding of the importance of sustainability practices. Given this a priori assumption, the analysis sought to identify whether the barriers and solutions identified by students and faculty were similar regardless of their adherence to or knowledge about sustainability, or whether those with more knowledge about or adherence to sustainability practices would identify different types of barriers for curriculum development.

#### *3.1. Student Responses*

#### 3.1.1. Student-Identified Barriers

Table 1 summarizes student responses identifying barriers to sustainability-based education at Cal Poly, including frequencies (total number of student interviewees N = 39). Any statement wherein a student identified a relevant barrier inhibiting their participation in sustainability-based education was coded as a Barrier. A single transcript could contain multiple coded barriers.

Though there is little variation between the "high" and "low" knowledge groups for the most frequently stated barriers in student responses, some interesting key results can be seen in Figure 1, which depicts the data graphically. The top three most frequently stated barriers in both "low" and "high" student groups are *Accessibility*, *Time Constraints*, and *Neglect*. *Accessibility* identifies barriers wherein students note they are unable to access sustainability-related courses. For example, as this student (second-year child development major, with low sustainability knowledge) shares:


**Table 1.** Student-identified barriers to sustainability-based education at Cal Poly and corresponding frequencies (Total number of student interviewees N = 39).

<sup>1</sup> LSK = Low Sustainability Knowledge; <sup>2</sup> HSK = High Sustainability Knowledge; <sup>3</sup> Values in columns three and four give the response frequency.

**Figure 1.** Distribution of percent effective responses corresponding to student-identified barriers to sustainability-based education at Cal Poly. Size of bubbles represent percentage of responses in each knowledge group. (Number of student interviewees N = 39; Total effective responses ER = 99. LSK and HSK indicate low and high sustainability knowledge, respectively).

*"I haven't really seen a lot of classes that focus on it, so I'm not enrolling in those classes because I don't know what they're about."*

*Time Constraints* represent a grouping of responses that represent an inability for the student to schedule sustainability-focused courses. As this student (second-year landscape architecture major with high sustainability knowledge) shares:

*"Time. I could actively seek out a place to educate myself if I had the time, but with classes and stu*ff*, but if it were a GE, I think we would have time to go, if it was in an actual class that I had to take, then I would be thrilled to have to go to it."*

*Neglect* represents responses where students share attitudes that represent a disinterest in the idea of sustainability, in general. For example, as this student (second-year electrical engineering major, with low sustainability knowledge) shares:

*"It's not really on my mind since it's not a problem that's currently a*ff*ecting me to a great degree..."*

Though the top three 'barriers' overlap between groups, those students with 'low' sustainability knowledge most frequently list *Neglect* as a barrier. In fact, *Neglect* is listed twice as often by 'low' as 'high' students. This indicates that *Time Constraints* and *Accessibility* are key barriers across student respondents, and those who lack sustainability-related knowledge may face a self-selection barrier beyond *Accessibility*.

Populating Table 1 data into Figure 1 suggests that most of the student participants appear to have a tendency to address barriers from personal experience, and few student participants can address barriers beyond the personal level. Figure 1 orients barriers on a continuum from personal to institutional level of experience. The bubble diameters display the percentage of responses in each knowledge group describing each barrier. The plot shows that responses derived from participants with low sustainability knowledge significantly skew toward personal level of experience and perception, whereas high-knowledge participant responses lean toward institutional observations.

It is not surprising that students with low sustainability knowledge also appear to neglect this subject, and vice versa. Other common barriers across student groups were mentioned much less frequently. These responses include statements that reinforce the general ideas that *Time Constraints* and competing priorities limit individual ability and/or desire to pursue sustainability-related education. These statements relate to lack of institutional or personal priorities, lack of motivation, and resource constraints (e.g., time, money, available electives). Although the result does not display significant variation, Figure 1 shows that students with better knowledge in sustainability tend to envision barriers from institutional aspects, whereas those with lower knowledge addressed sustainability from a personal perspective.

#### 3.1.2. Student Identified Solutions

Table 2 and Figure 2 summarize student responses identifying solutions to address barriers to sustainability-based education at Cal Poly and the corresponding frequencies. These were generally coded as part of responses to the interview question: "What are some ways to make sustainability education more accessible at Cal Poly?" Any statement wherein a student identified a relevant strategy to enhance participation in sustainability-based education was coded as a Solution. A single transcript could result in multiple solutions.

Despite the fact that several barriers were recognized based on student interviewees' personal perception (Table 1 and Figure 1), all solutions proposed suggested how Cal Poly should tackle the challenges from an institutional level (Table 2 and Figure 2). This discrepancy implies student participants collectively recognize institutional opportunities to promote sustainability yet are less willing to act or make a commitment at a personal level.

Under the "low" and "high" knowledge categories, the top three solution responses in terms of frequency of occurrence are: *Promotion*, *Integration*, and adding a *General Education Option*. Statements coded as *Promotion* include responses identifying the use of fliers, booths, or events to promote sustainability curricula. This result seems ironic, because Cal Poly organized and heavily promoted an Earth Week event during the week prior to our interviews. There is a significant disengagement between ignorance and the suggestion to "promote" sustainability. As a result, the authors are skeptical about the effectiveness of event promotion in overcoming sustainability barriers on campus.

**Table 2.** Student-identified solutions to improve sustainability-based education at Cal Poly and corresponding frequencies (Total number of student interviewees N = 39).


<sup>1</sup> LSK = Low sustainability knowledge; <sup>2</sup> HSK = High sustainability knowledge; <sup>3</sup> Values in columns three and four give the response frequency.

**Figure 2.** Distribution of percent effective responses corresponding to student-identified solutions to improve sustainability-based education at Cal Poly among low- and high-knowledge (LSK and HSK) student participants. Size of bubbles represents number of responses (Number of student interviewees N = 39; total effective responses ER = 81).

Though the top solutions from the 'low' and 'high' groups once again overlap, the 'high' group lists a greater number of solutions than the 'low' group. Additional responses include *Link to Major*/*Job*, creating *Activities and Events*, *Early Awareness*, and making it a *General Education Requirement*. The 'low' group also mentioned *Link to Major*/*Job* and Making it a *General Education Requirement*.

The ideas of *Integration* and *Linking to Major*/*Job* pose interesting and nuanced solutions derived from the transcript data. *Linking to Major*/*Job* represents responses where students suggest the importance of sustainability should explicitly link to how it will benefit their future job prospects and careers. For example, as this student (first-year political science major, with high sustainability knowledge) shares:

*"* ... *if you did it in the curriculum, not like literally taking a sustainability class, but it could be integrated into certain classes, because it's very applicable to di*ff*erent things."*

*Integration* captures responses where students suggest that instead of creating new courses or new requirements, sustainability education should integrate into already existing curricula, as this student (second-year engineering major, with low sustainability knowledge) notes:

*"I'm not feeling taking a required class just for sustainability. Maybe incorporate into classes that are already* ... *GEs that are already required."*

Required general education (GE) courses comprise approximately one-third of the total units for each degree at Cal Poly. Subjects include lower- and upper-division courses in: Communications, sciences and mathematics, arts and humanities, society, and technology. All references to the *General Education Option* category captures student responses suggesting that a viable solution would be adding additional sustainability-related courses to the curriculum as options for completing general education (GE) requirements. This is slightly more popular (judging from the responses) than the solution of adding an additional GE Requirement. A *GE requirement* would modify the curriculum across the campus to ensure all students complete a sustainability course, whereas the *GE Option* solution would provide students with the ability to fulfill a broad GE requirement by choosing to take a sustainability-related course. The requirement is a more rigid, yet broader ranging solution.

#### 3.1.3. Student Identified Barriers and Solutions by College

Although student responses for barriers and solutions were similar regardless of 'low' or 'high' sustainability knowledge, we note some deviation when examining responses by the six academic college units on campus (Table 3). Respondents span all colleges across campus in similar (though not identical) proportion to the make-up of the university. Due to the intensive nature of qualitative data analysis, sample sizes are small. Though our study reveals important and actionable data on barriers and solutions in sustainability education, the sample size falls short in making comparisons across different colleges. The sample has a lower representation of Science and Math students with high sustainability knowledge and generally few participants from the college of Business. Despite the small sample size, data did reach data saturation, which indicates validity of the overall findings [37]. However, we suggest only drawing tentative and university-specific conclusions from these data.



Despite these limitations, responses from students do reflect the characteristics of collegiate curriculum. For instance, Engineering students often identify *Time* and *General Education Options* as barriers and solutions because of discipline-specific constraints that limit the freedom of engineering students to pursue elective units outside their professional curriculum requirements. This has undoubtedly contributed to an institutional attitude of efficiency. Thus, one feasible approach to increasing sustainability education might arise by adding formal options to an already restricted curriculum.

Meanwhile, Liberal Arts students most frequently identify *Accessibility* and *Integrating* as barriers and solutions. This observation is interesting, because it may reflect the College of Liberal Arts' approach to a more integrated and holistic liberal arts education, despite the major within the college. Again, all conclusions are tentative given the sample size. However, results may indicate that discipline-specific solutions are needed to promote and improve sustainability education on campus.

#### *3.2. Faculty Responses*

#### 3.2.1. Faculty-Identified Barriers

Table 4 summarizes faculty responses to identifying barriers to sustainability-based education at Cal poly, including frequencies (N = 17 faculty interviews). As apparent in Table 4 and Figure 3, the barriers noted by faculty differ considerably from those identified by students. In addition, faculty responses show variation depending on sustainability knowledge.

**Table 4.** Faculty-identified solutions to improve sustainability-based education at Cal Poly and corresponding frequencies (Total number of faculty interviewees N = 17).


<sup>1</sup> LSK = Low sustainability knowledge; <sup>2</sup> HSK = High sustainability knowledge; <sup>3</sup> Values in columns three and four give the response frequency.

**Figure 3.** Frequency distribution of faculty-identified barriers to sustainability-based education at Cal Poly among groups with different sustainability knowledge levels (Number of faculty participants: N = 17; effective total responses: ER = 46).

Similar to the pattern observed in Figure 1, faculty members with low knowledge level describe barriers from a personal level, whereas high-knowledge faculty groups address the challenges across

the wider spectrum of aspects (Figure 3). Faculty members with 'low' sustainability knowledge cite a *Lack of Knowledge* or *Lack of Awareness* as major barriers towards advancing sustainability across the curriculum. The attitude of neglect or carelessness in sustainability appears to be a noticeable driving force determining the perception of teaching sustainability among faculty participants with 'low' sustainability knowledge. Less frequently mentioned by 'low' knowledge faculty are *Discipline Restrictive*, *Priority*, *Accessibility*, and *Time*. 'High' knowledge faculty emphasized *Discipline Restrictive* and *Priority*. The theme of *Discipline Restrictive* intends to capture responses from individuals who note that the subject matter of a class or discipline can restrict the integration of sustainability themes. For example, as this faculty member (associate professor in the Mathematics Department, 12 years, with high sustainability knowledge) notes:

*"In some courses, it's more natural to fit in than others. When you're teaching students how to do calculus, you don't need to know about sustainability to do the technical thing, but you can include those topics* ... *It's important to include in topics. Some topics are easier to include than others."*

The theme of *Priority* generally refers to statements indicating that sustainability is not prioritized across certain curricula. This theme echoes student concerns regarding their own time (e.g., time to graduate, time for additional units). From a faculty perspective, this theme refers to institutional priorities for curriculum development. The *Priority* theme differs from those responses coded as *Time*, which refers to the ways in which faculty prioritize their own time in light of competing priorities. For example, a professor whose response was coded for *Time* might not choose to prioritize spending her time developing sustainability curricula. This observation is captured in the following response from a faculty participant (professor, Statistics Department, 18 years, with low sustainability knowledge):

*"Especially me who's been here forever, I've been here a long time. I don't always change".*

#### 3.2.2. Faculty Identified Solutions

Faculty ideas for barrier-specific solutions to improving sustainability-based education were revealed during the interviews when the faculty were asked how identified barriers might be overcome. Table 5 and Figure 4 summarize the faculty-identified, barrier-specific solutions by 'high' and 'low' sustainability knowledge.


**Table 5.** Faculty-identified solutions to improve sustainability-based education at Cal Poly (N = 17 faculty responses).

<sup>1</sup> LSK = low sustainability knowledge; <sup>2</sup> HSK = high sustainability knowledge;<sup>3</sup> values in columns three and four give the response frequency.

As is apparent in Table 5, faculty solutions varied depending on sustainability knowledge. For example, those faculty members with 'low' sustainability knowledge suggested *More Faculty Training* to help to incorporate sustainability themes in the classroom. Other solutions include *Promotion* and *Integration*. Those faculty members with 'high' sustainability knowledge suggest that solutions or improvements are the responsibility of the institution, (*Institutional Responsibility*), which might also include a responsibility for providing increased training to identify those classes that could most easily integrate sustainability-related themes and/or course buyouts to allow for curriculum development. Similar to students, faculty in the 'high' knowledge group also frequently express *Integration*, *Link to Major*/*Job*, and *Early Awareness* as a potential solution for overcoming barriers. Considering together the faculty solutions of *More Faculty Training*, *Institutional Responsibility*, and *Integration* into curriculum could provide a blueprint for how universities might advance sustainability education across the curriculum in a way that not only achieves institutional goals related to sustainability education but does so in a way that enhances the desire to teach and learn about sustainability. Moreover, similar to the discrepant pattern between identified barriers and solutions, personal commitment to take action remains questionable.

**Figure 4.** Distribution of percent effective responses corresponding to faculty-identified solutions to improve sustainability-based education at Cal Poly among low- and high-knowledge (LSK and HSK) faculty groups. Size of bubbles represents number of responses (Number of faculty interviewees N = 17; Total effective responses ER = 36).

#### **4. Discussion—Designing Barrier-Specific Solutions**

It is well established that interdisciplinary and non-siloed approaches are crucial to the incorporation of sustainability science into the curriculum [10]. However, details of how disciplines work as a system to contribute to sustainably curriculum are less apparent. These results reveal that gaps in the higher education system limit the expansion of an interdisciplinary sustainability curriculum. Findings from this study indicate that both students and faculty with higher sustainability knowledge have the tendency to identify barriers related to institutional accountability. A former study can well support the importance of this aspect, in which its authors found that an institution's internal rules and setting can be the driving force shaping people's behavior and decision-making processes [39]. This is particularly important for the development of solutions, as an institution will need to initiate a holistic strategy to incorporate sustainability into its core values. Being part of the ongoing effort, Cal Poly has incorporated several institutional measures, such as the creation of sustainability learning outcomes for all graduating students. It is important to assess the alignment between institutional measures and actual barriers that need to be addressed.

Noticeably, certain solutions for a singular barrier might be confronted by additional barriers. This is expected because these solutions reflected interviewees' intuitive rationalization, rather than those proposed based on thorough and systematic assessment and reasoning. Similarly, this is also why not all the proposed solutions were ranked from individual to institutional levels (Figures 2 and 4) as was done to identify barriers (Figures 1 and 3). In these cases, all the participants demanded institutional actions, rather than identifying what actions they would be willing to take. This can imply future challenges to make substantial impacts at an individual level even if Cal Poly can implement campus-wide solutions to encourage practicing sustainability. Recognizing personal responsibility to engage in learning sustainability needs to be addressed as one of the key steps to either promote a top–down or bottom–up approach to facilitate learning and teaching in sustainability. In the meantime, we also view this challenge as a new opportunity for establishing the momentum to promote sustainability education. A former study states that one of the possible tactics to improve personal awareness in sustainability is to create a "cognitive dissonance between individuals' values and behaviors" [40]. Therefore, findings from our study can provide ready input to address the dissonance between recognition of challenges and willingness to engage in making changes at a personal level, especially among the groups with lower sustainability knowledge.

Other studies note that expanding sustainability education and behavior must also address personal concerns and take into account increasing awareness of issues, messaging solutions, increasing convenience, and incentivizing change [41]. This resonates with the study's initial understanding of barriers, that the design of specific solutions may be best supported by institutional initiatives and availability to resources. Future areas of study will seek to better understand the role of faculty programs in the design and incorporation of barrier-specific solutions in higher education. As part of an effort to develop solutions to faculty-identified barriers, the authors will pursue a program of solutions, many of which could provide incentives in the form of modest stipends and opportunities for professional development and publication that begin to address identified barriers and solutions. These include:


#### **5. Conclusions**

A well-known core principle of sustainability education holds that interdisciplinary approaches are crucial, but the details of how these disciplines work as a system to contribute to an overarching sustainability curriculum are less apparent. These results reveal the gaps in Can Poly that limit the expansion of an interdisciplinary sustainability curriculum, which can resonate with institutes with similar academic setting. Cal Poly's experience indicates that the implementation of a sustainability-related curriculum must rely on multidimensional strategies and approaches. Recognition of barriers ranging from the personal to the institutional level will aid in the design and implementation of any expanded sustainability-related curriculum or program initiatives. Research results confirm that barriers to sustainability education exist across disciplines, participants with varying levels of sustainability knowledge, and among student and faculty groups.

The results from both students and faculty show that participants with higher sustainability knowledge have the tendency to identify barriers to sustainability education by holding organizations accountable, while participants with lower sustainability knowledge have the tendency to identify barriers to sustainability education by holding themselves accountable. This finding well depicts the mentality of students and faculty in supporting sustainability education. The finding also indicates that a campus seeks change to include sustainability education must initiate a holistic strategy to incorporate sustainability into its core values from the institutional level, with which individual awareness can be better promoted. This study's results confirm both approaches will be required to address perceived barriers to implementation. Continued research and understanding of the factors impeding the implementation of sustainability education could help students, faculty, and institutions to develop those holistic strategies.

**Author Contributions:** Conceptualization, all authors; Methodology: B.P., Y.-W.C., and D.N.; Writing—original draft preparation, B.P., Y.-W.C., D.N., and R.O.; Writing—review and editing, all authors; Visualization, Y.-W.C.; Green Campus Team supervision, K.S.; Funding acquisition, B.P., R.O., and Y.-W.C.

**Funding:** The APC was funded by USDA NIFA, Award No: 2017-70003-26380.

**Acknowledgments:** We thank Patrick Sullivan, Director, Center for Teaching, Learning & Technology, Cal Poly State University San Luis Obispo for his role in organizing the learning community where this work originated and continued support throughout the project. We acknowledge the Green Campus Team's participation in data collection via focused interviews.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **References**


© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

*Article*

## **Sustainability and Indicators of Newly Formed World-Class Universities (NFWCUs) between 2010 and 2018: Empirical Analysis from the Rankings of ARWU, QSWUR and THEWUR**

## **Zhimin Liu 1, Goodluck Jacob Moshi 2,\* and Cynthia Mwonya Awuor <sup>2</sup>**


Received: 25 February 2019; Accepted: 10 April 2019; Published: 14 May 2019

**Abstract:** In the 21st century, sustainability and indicators of world-class universities have come within the scope of an academic cottage industry. The complex problem of university sustainability implies a big challenge for countries and educators to implement important strategies in an integrated and comprehensive way. This paper highlights and analyzes the sustainability indicators of universities included as newly formed world-class universities (NFWCUs) in the top 100 from 2010 and 2018. The integration of three global ranking scales—the Academic Ranking of World Universities (ARWU), the Quacquarelli–Symonds World University Ranking (QS) and the Times Higher Education World University Rankings (THEs)—allows us to minimize the impact of the methodology used. This study integrates regression analysis by using statistical grouping, case studies and normative analysis. Our principal findings are as follows: among the commonly ranked top 100 universities in 2018, the ARWU, QS and THE counted 57, compared with 47 in 2010. Thus, comparing 2010 and 2018 shows that 44 of the universities appeared simultaneously in ARWU, QS and THE rankings and maintained a sustainable position in any ranking system in the family of top 100 groups. Three lower-ranked NFWCUs in the hybrid list for 2010 lost their ranking and did not appear in the group of top 100 universities in 2018, which are covered by some catch-up and young universities. The NFWCUs were from US, Australia, China, Singapore, Germany and Belgium. By systematic comparison, the US and UK continued to dominate the stability of NFWCUs in 2010 and 2018. The key sustainability indicators include a high concentration of talent, abundant resources to offer a rich learning environment and conduct advanced research. Generally, the factors were negatively associated with ranking suggesting that a higher score result in top ranking and vice versa. Teaching, research, citation and international outlook were negatively correlated with THE ranking in 2018. Similarly, Alumni and PUB were negatively associated with ARWU ranking in 2018. All factors except international student ratio were significantly correlated in QS ranking either in 2010 or 2018, where negative association was observed. The significant contribution of our study is to highlight that for the sustainability of universities, it is necessary to have an increasing emphasis on the effectiveness and efficiency of government-supported research, stability of investments and more approaches to employ international initiatives. The results also confirm the appropriate governance, developing global students and place emphasis on science and technology as additional factors in the approaches of pathways to NFWCUs, with delivery of outstanding educational programs and comprehensive internationalization as a key indicator for performance improvement and global university ranking systems.

**Keywords:** world-class universities; newly-formed; sustainability; rankings; indicators; initiatives

#### **1. Introduction**

Over the past few decades, the concept of world-class universities (WCU), also called globally competitive, elite, early formed or traditional universities, has emerged. At present, this term has become a catchphrase, not only in terms of improving the quality of learning and research in higher education but also for sustainability and developing the capacity to compete explicitly in the global higher education marketplace [1–5]. Consequently, since the appearance of global university rankings in 2003, the goal of establishing and changing universities to WCU is to be able to compete in the global knowledge economy [6,7]. Furthermore, the aim is to train creative human resources and advance national development through the acquisition, adaptation and creation of an advanced knowledge-based economy. However, in many ways the development of higher education has facing international and domestic pressures. In today's globalized world, several countries have removed hurdles to join a global battle to establish their institutions as WCU and to promote the knowledge-economy society [8,9]. At the same time, establishing world leading position has been ambition of both government and top universities.

Meanwhile, countries in the western world, particularly the United States (US) and the United Kingdom (UK) and several Asian countries are launching programs for university authority, structure, system and organizational goals to enhance the sustainability and competitiveness of their universities. According to Hezelkorn and Li, assuring world-class university status is the wish for every nation [8]. Subsequently, a variety of reforms and development strategies at both the national and institutional level have been defined and observed. These reforms have also been strengthened and intensified with the propagation of international league tables [7,10–13]. According to Mok, to help their universities achieve this exclusive status, many countries and regions have also implemented a number of funding initiatives.

Generally speaking, high-profile university excellence initiatives are becoming increasingly popular research and development policies globally. These initiatives were institutionalized in more than 28 nations from 1995 to 2017, including in Europe, Latin America and Asia. Such initiatives include Germany's Excellence Initiative; Japan's Centers of Excellence for the 21st Century; South Korea's BK21; China's 211, 985 and 2.0 Projects; Singapore's Research Centers of Excellence; and Australia's ARC Centre of Excellence. The aim is to help their countries and elite universities achieve global competitiveness and higher-quality colleges and universities [14,15]. Again, according to Mok and Salmi, higher education worldwide has been experiencing a continuous trend of transformation shaped by different types of international drivers which operates in a constant flux of globalization. With the strong intention to rank highly in global university leagues, governments are exerting serious efforts to boost their universities global competitiveness. The goal for establishing WCU and changing existing universities to WCU is to be able to compete in the global knowledge economy [16]. Thus, it is important not only to improve the quality of research and teaching but also more significant to develop the competency to compete in the global market economy [17,18]. The term "excellence" has gained in relevance in recent years to sustainability of higher education which are oriented to international ranking criteria. Given this, the quest for WCUs can be understood as an institution that transcends culture and education [19]. They are a "point of pride and comparison among nations that view their status in relation to other nations" [18,20]. In this sense, developing WCU is both a national and local strategy in a worldwide context and sensitive to universal referents and objectives. Several reports confirm that there is no clear definition and statement about the context of what constitutes a world-class university [9,14,21]. Three complementary sets of dimensions of action are in play and can be found in most NFWCUs in the last eight years: favorable governance features that encourage strategic vision, innovation and flexibility; abundant resources to offer a rich learning environment and conduct advanced research; and a high concentration of talent [22,23].

Indeed, all universities are desirous of achieving world-class status; however, the paradox of WCU, as Altbach has succinctly and accurately observed, is that "everyone wants one, no one knows what it is and no one knows how to get one." Thus, while the goal of WCU status is clear the definition of world-class status is not. Therefore, a set of complementary factors are needed to define the sustainability of NFWCUs [24,25]. In line with the work of these authors, the aim in paths of governance to transform WCU is rooted not just in thoughtful reflections but also in the symbolic role of such higher education institutions (HEIs). The facts tell us that the path for WCU is not something that we can fix overnight. In addition, it is not something that universities can proclaim themselves, as Salmi point out: "becoming a member of the exclusive group of WCU is not achieved by self-declaration; rather, elite status is conferred by the outside world on the basis of international recognition" [5,11]. We have to learn from the institutional characteristics of both early and newly formed WCU as a starting point [26,27]. The global university rankings make competition between states visible and thus are commonly recognized as indicators of success due to excellence-driven policy goals [28]. Hence, strong engagement toward becoming WCU is also a trend nowadays.

Since the 2000s, a new era in HEIs is characterized by global competition in which university ranking systems have assumed importance. Since the emergence of global university rankings in 2003, the interrelated connection between WCU and university rankings has been a heated topic around the world. Thus, global university rankings have become a useful measure of university competitiveness and are commonly used to undertake quality assurance of higher education systems, their strength and weakness, internal analysis, academic performance of top-ranked higher education and their variations [29,30]. It has been found that ranking systems often serve as a proxy for world- class status and despite of many issues with rankings, most are driven by the purpose and concept of university quality. Various methodologies are used in these ranking systems and their influence is striking. Ranking has become unavoidable and it will remain part of academic life.

The ranking of universities has affected HEIs, as ranking appears to strengthen or grant visibility to some universities and expose perceived challenges at the system and institutional level [19,31–33]. Ultimately, the governments of several countries have made a strategic choice for relative targets and formed new visions and missions, university structures and functions, thus eventually turned to lead HEI policy change from the national system level to form top-ranked universities [6,34,35]. According to Teodoro (2019) the compilation and use of university rankings is a widely debated issue. There was some previous interest in this field: in 2003 the Academic Ranking of World Universities (ARWU) by Shanghai Jiao University was published and then the National Taiwan University Ranking (NTU), the University Ranking of Academic Performance (URAP), the Centre for Science and Technology Studies, Leiden Ranking and Scimago Institutions Rankings. Other rankings take into account dimensions not exclusively related to research data, as they incorporate opinion surveys, such as the Times Higher Education World University Rankings (THE) and the Quacquarelli–Symonds World University Ranking (QS). The Multidimensional Global University Ranking (UMultitank) endeavors to resolve some of the criticism commonly leveled at the above rankings.

Rankings have been the object of much criticism [16,19,36]. Composite rankings have numerous weak points, as indicated by several authors. There are numerous major global and national ranking systems that describe the performance of universities in the world; among them, we chose three prominent systems: the Academic Ranking of World Universities (ARWU), the Quacquarelli–Symonds World University Ranking (QS) and the Times Higher Education World University Rankings (THE), which are the most cited and commonly used [18,37]. World rankings aspire to include the most relevant universities worldwide. Thus, composite rankings summarize several weighted indicators and assign one score, which is then used to offer a classification. Rankings should be used with caution, taking into account how they are developed [38].

#### *Approaches to Common Composite of World University Ranking Systems*

Rankings provide great insight into the strength and shifting fortunes of individual research universities, although the compilation and use of university rankings is currently a widely debated issue. University rankings, despite their limitations, have been considered to be important by many stakeholders [37]. It is evident that many, though not all, institutions of higher education take rankings

seriously for the purpose of accountability, evaluation and strategic planning. At the individual level, rankings are potentially useful in providing a comparable and clear summary of information for students to select appropriate universities and destinations for their education. At the institution level, global rankings have become a marketing feature in university activities [39]. Rankings strongly influence the behavior of universities, as their presence in ranking tables alone heightens their profile and reputation, encouraging the collection and publication of reliable national data on higher education. Internationally, rankings provide an informal measure of a country's ability to compete in a knowledge-based world economy due to the emphasis on research output. Therefore, several authorities and universities have put forward policies for creating WCU and take university rankings seriously. According to Pusser and Marginson, rankings are an essential sustainability instrument for the exercise of power in the service of dominant norms in global higher education.

It is clear that rankings continue to grow in popularity and gain interest by policy makers. While the world is obsessed with rankings, there are several cautions voiced against them. Rankings have been the object of much criticism with regard to the measurement indicators, the institutions being measured and the diversity of features to make comparisons [40]. A consensus has emerged and there are plenty of discussions in the literature that despite the wide diffusion of university rankings in recent years, their methodology and usefulness are not exempt from critics, especially the validity of the results and suitability of the variables used. Several studies have also criticized that the selection of indicators and their weighting, data processing and transparency are strongly questioned. Again, every ranking is based on the availability of comparable data and is built on the subjective judgment (over indicators and weightings) of its compilers. In addition, the indicators used promote the presence of a minimal number of research institutions from peripheral countries and neglect some other types of college and universities. [7,29]

University ranking systems provide a comparative analysis of university performance and characteristics at a global level, regional level (e.g., Europe, Latin America), national level (e.g., US), development level (e.g., developing countries) and particular group level but several ranking systems are available for several purposes and only a few attract many and are commonly recognized. In line with the above, this study considers the three rankings of ARWU, QS and THE due to their reputation, characteristics, influence and limitations. There are more than 28,077 (http://www.webometrics.info/ en/node/54) HEIs all over the world and the top 100 universities account to 0.004%, which should be recognized worldwide due to their sustainable performance indicators and academic excellence (www.topuniversities.com). This paper addresses the gap in sustainability indicators of NFWCUs in the last eight years. Then, if we agree that any universities listed on the top 100 in the global ranking systems of ARWU, QS and THE as world-class, this research will define those that have fresh appeared as newly formed (or progressive-type) world-class universities (NFWCUs).

#### **2. Research Objectives and Goals**

The general goal of this study is to examine sustainability indicators and establish present evidence on the empirical distribution of and contributions across NFWCUs in the last eight years according to the top world ranking systems of ARWU, QS and THE. In today's globalized world, higher education institutions worldwide are in a period of difficult transition and transformation shaped by different types of internal drives. This situation is mostly affected by globalization, the advent of massification, undefined relationships between the state and universities and current technologies, among other factors [41]. In this view, an increasingly pressing agenda and priority of many governments is to make sure that their elite universities are operating at the cutting edge of intellectual and scientific development [42]. In the 2000s, several countries joined the global pursuit of building WCU in the knowledge society [43,44] and currently, several countries are creating WCU as an essential part of their higher education reform [45]. In recent years, building a sustainable world-class university has been the dream of some nations and emerging approaches have attracted attention from scholars. However, up to now few countries have had the possibility to turn the dream into reality.

According to Peter Senge, within the unpredictable future business environment and accelerated development of the knowledge economy, universities need to increase their knowledge generation and knowledge transfer in society. Universities should strive to become learning organizations [46–48]. They must enlarge their focus on research and their traditional mission of teaching and learning. Universities should adopt a new paradigm to monitor the needs of different stakeholders. This means they should create adaptation knowledge to produce generative knowledge and to become learning organizations and at this stage the government should become a strategic driving force [3,48] for universities and a powerful integrator. Since all main functions of a university are related to knowledge transformation and distribution, the university becomes a knowledge-intensive organization dominated by intellectual capital [49].

Since the beginning, the university has always been a cultural and moral symbol of social communities. The perspectives, preoccupations, activities and goals of universities have significantly changed over time, as have their roles and strategies. Nowadays, universities are viewed as knowledge providers, innovation facilitators, promotors of entrepreneurial talent, economic and civic leaders and mostly knowledge pioneers of the creative commons [49]. Countries around the world have faced and continue to face challenges in global university ranking competitiveness and the increased importance of having top listed universities.

Higher education governance continues to grapple with increased competition under pressure from the curriculums they offer and the university ranking systems used. In developing countries, higher education faces several challenges, such as inadequate funding, outdated curriculums and governance structure. Whereas many developed countries have made changes in their higher education system to deal with the growing pressure of global competitiveness of universities across the world, developing countries continue to lag behind [50]. In many developing countries, the lack of reformation and innovative changes in management practices and governance models in higher education is stifling high-quality teaching, reputation, productive faculty, excellent students, flexible administration, full funding and internal engagement.

The literature indicates that WCU involves a complementary set of three factors. These factors can be observed in most top-ranked universities. Typically, these factors are a high concentration of talent among both faculty and students; abundant resources to support a rich learning environment and conduct advanced research; and constructive governance features that encourage strategic vision, innovation and flexibility to enable institutions to make autonomous decisions and manage resources without being burdened by bureaucracy [27].

In the rest of the world, world-class universities have in general kept pace with reforms in governance structures and academic systems. A number of researchers [46,51,52] have analyzed the contributions of university governance and educational systems to a world-class reputation. Most handful of recent scholars indicate that the key determinants that have a strong influence on the development of world-class universities include excellence in research, productive faculty, excellent students, flexible administration, adequate facilities, full funding and international engagement. According to the definition of world-class universities by Philip G. Altbach, the criteria include governance structure and productive faculty. Therefore, when we try to understand the paths of governance for NFWCUs, we must keep in mind that each nation is unique, and its universities can survive and prosper if they meet the needs of the society.

In the 21st century, the top universities are under tremendous pressure and have upgraded to world research once in a relatively short time. They have made many policies at unprecedented speed. Other universities and their governing agencies have also set such an aim. The movement for NFWCUs has become a national role, with the investment of many resources. The question is, can universities transform into NFWCUs only with a strong will and high financial investment? The answer is unknown. In line with the criteria above, the central objective of this study is to assess the driving indicators for variations and contributions of NFWCUs based on characteristics and

distributions among countries. Further, the study suggests common measurable strategies applied by these universities to achieve success.

#### **3. Research Questions and Methodology**

To guide our inquiry related to this paper and reach the main findings, this study integrates research methods using statistical groupings, case studies and normative analysis. The following research questions were used to guide our study:

RQ1: What are the newly formed world-class universities (NFWCUs)?

RQ2: What are the common shared sustainability indicators of NFWCUs?

RQ3: What are their variations and contributions in the future?

RQ4: What are the research productivity and government initiatives? Specifically, how have these governments and HEIs construct and transformed to become globally competitive.

Although there still are no generally accepted definitions or clear standards for quantified assessment of world-class universities in the world, there are various indexing systems for assessing universities. Each has its own motives and purpose as well as shortcomings and limitations. It is therefore a focus on being world-class universities has become synonymous with relative efforts by universities to move upwards in the international league tables. Thus, league tables are taken as symbolic and powerful indicator to prove striving to NFWCUs.

To achieve the aims of the research questions and develop explanations of NWFCUs as a social phenomenon, according to Liu (2015) and Teodoro (2018) this study adopted a qualitative research design. Qualitative research entails opinions, feelings and experiences. It aims at describing social aspects as they occur and taking a holistic approach, thereby extricating critical experiences from NFWCUs and facilitating the optimization of excellence sustainability and indicators in the last eight years by providing suitable suggestions. According to Pusser and Marginson (2013), rankings are an important element in global higher education. In our current study, we use three known rankings with their relative indicators. By synthesizing these rankings, NFWCUs are obtained. This reduces the heterogeneity of single rankings and weaknesses as indicated by Soh (2017b). Keeping in mind that every ranking has some statistical problems, this study implements and integrates four research methods: case studies, statistical groupings, comparative analysis and normative analysis. The integration of these four research methods with the global ranking systems of ARWU, QS and THE provides an opportunity to build a greater assortment of divergent views on the issue studied, increase the credibility of the study and offer stronger inferences. Taking this into consideration, this is the important reason why this study integrates four research methods.

#### *3.1. Methods of Data Collection*

Our study retrieved data from three databases that are most publicly visible and well recognized globally for university rankings: ARWU, QS and THE, in which the factors used to determine university rankings are provided. It was evident that each ranking system has unique factors, hence this study compared university rankings among the three databases. To be included in the study, a university must appear among the top 100 universities in all three ranking systems. The universities were selected from 2010 and 2018.

The first step is to obtain data from ARWU, QS and THE using the values of indicators they use and global scores; after that, we input the top 100 universities that appear in all three ranking systems. Finally, universities with all characteristics are identified and analyzed by a regression model.

We selected NFWCUs according to a set of relative criteria to find current NFWCUs from the last eight years (2010–2018) based on the ARWU, QS and THE ranking systems. We then combined the current data from ARWU, QS and THE on the top 100 universities ranked previously by using their summation and total mean so as to know the order of overall top-ranked universities and a hybrid list of NFWCUs. Finally, the overlapping part was synchronized to drive classifications, disparities and distributions among countries.

It should be noted that all university ranking information related to institutional practices are available from the web pages of the three ranking systems: http://www.shanghairanking.com/, https://en.wikipedia.org/wiki/ and the NFWCUs' home pages. By considering the adopted system, the four methods were used to integrate the three ranking systems to produce a hybrid list of current NFWCUs in the last eight years. The timeframe appeared to be a period during which the governance of several universities-initiated excellence initiatives. Moreover, such an approach allowed us to deal with the criticism of inconsistent ranking results because of different methodologies used by the three selected systems; if we simply look a single ranking list, it is not enough to fully judge the quality and level of a university, so it seemed that integrating the top three ranking systems to evaluate the sustainability indicators of NFWCUs would be more comprehensive for this study. Following the above-explained methods, a summary of the process and the population of the study and sample design are illustrated as follows.

#### 3.1.1. Academic Ranking of World-Universities (ARWU)

The Academic Ranking of World Class University (ARWU; www.arwu.org) undertaken by Centre for World-Class Universities of the Institute of Higher Education of Shanghai Jiao Tong University, China (hence often known as Shanghai Jiao Tong Ranking). It was firstly started in 2003 as the first world global ranking of universities. Thus, the origin was to establish the global standing of top Chinese universities but soon attracted world attention. This ranking has attracted much interest from around the world as it ranks the best world's 500 top universities from 41 different countries annually. Since 2009, the Academic Ranking of World Universities (ARWU) has been published and copyrighted by Shanghai Ranking Consultancy, a fully independent organization on higher education intelligence. Although the initial purpose of ARWU was to find the global strength standing of top Chinese universities, it has attracted a great deal of attention from universities, governments and public media worldwide. Moreover, a survey on higher education published by The Economist in 2005 commented ARWU as the most widely used annual ranking of the world's research universities. The significant influence of ARWU is that its methodology is scientifically sounds, stable and transparent. Its content is widely cited and employed as a starting point for identifying national strengths and weakness. [36,45,53]. Being as the first digital instrument of its kind, it is one of the causes of some of the key features of global academic competition as we know it today.

#### 3.1.2. Quacquarelli-Symonds World-University Ranking (QS-WUR)

The Quacquarelli-Symonds World-University Ranking (QS; www.topuniversities.com) is an annual publication of university rankings. Previously known as Times Higher Education-OS World University Rankings, being similar in many ways; this is a global career & education company specializing in education and study abroad. From 2004 to 2009 THE and QS jointly published the same rankings. The fundamental criteria used were research, employability, teaching & internationalization. However, in 2010, after separating from THE, QS continued with virtually the similar criteria for its annual rankings, on the whole with some changes in the weight of the requirements ranks about 700 of the world's top universities [33,54]. Being the only international ranking to have received International Ranking Expert Group(IREG) approval, the QS is viewed as one of the three most-widely ready university rankings in the world [55,56].

#### 3.1.3. Times Higher Education World University Ranking (THE-WUR)

Times Higher Education World University Ranking (THEs; www.timeshighereducation.com/). Is an annual publication of university rankings. Actually, as early as 2004 Times Higher Education (THE) partnered with Quacquerelli Symonds (QS)-QSWUR to publish a new set of world university rankings. However, by in 2010, THE & QS ended their partnership, as each one deciding to release

its ranking with two independent programs. In 2010, THE using new data supplied by Thomson Reuters (a business data provider headquartered in New York) published its rankings using a different methodology. THEs currently uses 13 performance indicators grouped in 5 areas of indicators, most of the data that are being provided by the institutions. Thus, ranks world's 400 top universities annually comprise of the world's overall subjects and reputational rankings. THE is often considered as one of the most widely observed university rankings and praised for having a new, improved ranking methodology. THE ranking is criticized however on having and relying on subjective reputation survey. [53]. Table 1 provides details on the indicators and weight for each three ranking methodology used in this study.


#### **Table 1.** University ranking methodology and sustainability indicators.

#### **4. The Conceptual NFWCUs Context and Analytical Framework**

References to the sustainability of NFWCUs immediately imply allusion to catch up, the most progressed type, the prominent among the easily formed within the state or national higher education institutions. The NFWCUs constitute almost universally within the English-speaking countries, post-secondary institutions that constitute aiming the pinnacle of either a state or national higher education system, of those have overlapped to excel, among others. This understanding usually depicts the largest and most elite huger regarded universities within the countries and a more substantial set of commonly acknowledged universities [57]. These can be seen as, designation, visibility, differentiation, validity and the various discourses around NFWCUs in Asia and Latin American higher education institution context were vocal critics analyzed in Reference [2]. These can be perceived as several Asian countries drive to the formation of WCUs.

Expounding on different phrases of "prestigious institutions" as a research university, "world class" and "Newly Formed World-Class Universities" (NFWCUs): Altbach, [2] articulate world-class universities as leading universities in the countries, largest producer of graduate students, research and publications. They are held high national esteem and play an important role in national capacity building and innovation effort and commonly included in the top-ranked universities. In century ago, only well-deserved universities such as Harvard Universities and Oxford University that were recognized as "Early Formed World-Class Universities" "EFWCUs." However, recently, several

universities are catching up to future the quest and catch up in of top 100 global university rankings. Invariably, in this study, the word "NFWCUs" used to mean, universities which are "fresh-appeared" "catching up" or "progressive type" are described based on the commonly ranking list of top 100 in the world's top three ranking systems. They are universities newly listed in the current top 100 ranked by the global university ranking of ARWU, QS and THEs in the past eight years (2010-2018). This broad conceptual framework of NFWCUs as enunciated here is deemed relevant for this study and adopted for the study of global NFWCUs. Therefore, early universities and NFWCUs shed some light on the sustainable worldwide movement to create WCU. By tracing the university ranking systems, the NFWCUs has a global phenomenon of strong reputation and influence on the development of higher education including productive faculty, excellent students, flexible administration, international engagement and plentiful funding mechanisms.

Although, the discussion as regards to developing WCUs involves integrated part assessment of teaching and research in global world rankings. In this paper, variation refers to the sequence with changes divergence in the range of sequences in global rankings of the WCUs regarding characteristics, data and functions. Since there are over 28,077 distribution and variations of colleges and universities ranked all over the world but the top 100 worldwide listed institutions in the three global ranking system are very few (cover around 0.004%). Therefore, there are certain limitations in the rankings of various universities and they are widely criticized, whether the top universities in the rankings must be the world's top universities still have doubts. However, in different university rankings, based on different ranking indicators and weight, have steadily ranked in the world for many years due to their institutional practices, characteristics, visible achievements and superior performances in the commonly included top higher education institutions (HEIs). Following this notion and by considering the world common ranking systems (ARWU, QS and THEs) in this paper the variation of the NFWCUs will mean disparities among the list of top-ranked HEIs in the global ranking systems from the last eight years (2010 and 2018) respectively. The NFWCUs contributions entail the role, impact and achievements in educational, social, economic and cultural arena in their countries—the literature on variation and contribution of NFWCUs in particular and universities. Today, there is a global debate on the importance of WCUs as a critical element for a quality generation in skills for a knowledge-based economy. Critical to national competitiveness and sustainable development in today's competitive global economy—NFWCUs described as the core, key and forces to build an inclusive and diverse knowledge society the subject of this paper. Figure 1: then summarises and indicates the conceptual framework of the study and showing the sustainability indicators in the three rankings.

**Figure 1.** Conceptual Frame Work.

*Data Analysis Tool*

The data was analyzed using computer program-Statistical Package for the Social Sciences (SPSS v22.0). Then, the regression analysis for overall sustainability and indicators affecting university ranking from ARWU, QS and THE was based on the following regression equation:

$$\text{ARWU} = \beta 0 + \beta\_1 \text{ \* (alummi)} + \beta\_2 \text{ \* (award)} + \beta\_3 \text{ \* (HiCi)} + \beta\_4 \text{ \* (N\&S)} + \beta\_5 \text{ \* (PUB)} + \beta\_6 \text{ \* (PUB)}$$

$$\beta\_6 \text{ \* (PCP)} + \text{SE}$$

QS = β0 + β<sup>1</sup> \* (ACADREP) + β<sup>2</sup> \* (EMPREP) + β<sup>3</sup> \* (STUDFAC) + β<sup>4</sup> \* (citations) + β<sup>5</sup> \* (INTFAC) + β<sup>6</sup> \* (INTSTUD) + SE

$$\begin{aligned} \text{THE} &= \beta 0 + \beta\_1 \text{ \* (teaching)} + \beta\_2 \text{ \* (research)} + \beta\_3 \text{ \* (citations)} + \beta\_4 \text{ \* (INIDINC)} + \epsilon\\ &\quad \beta\_5 \text{ \* (INTOUT)} + \text{SE} \end{aligned}$$

Key: β0 = constant; β1, β2, β<sup>3</sup> ... = regression coefficient; SE= standard error

Figure 2 Summarizes and indicates detailing steps and the process of data, raking indicators, characteristics and analysis process of the data, where by data grouping, normative analysis, case study and regression analysis appeared to reach the results of weightings on factors affecting ranking.

**Figure 2.** Summary of the process of data collections.

#### **5. Results of the Study**

#### *5.1. Sustainability and Indicators of NFWCUs in the Last Eight Years*

To address the research question, Tables 2 and 3 summarize the latest findings of the ARWU, QS and THE, comprising data from 2010 and 2018. Taking into consideration the hybrid list of top 100 universities in each ranking, the comparative analysis shows that there were only 47 NFWCUs in 2010. However, the number increased, reaching 57 in 2018. In addition, taking the hybrid list of 2018 NFWCUs, the figure indicates that 44 universities continuously maintained their status among the top 100 in the past eight years but three declined in rank and dropped out of the NFWCU family in 2018: Ecole Normale Supérieure in France and Brown University and the University of Minnesota in the United States. As for 2010 NFWCUs, the trend shows that 13 universities appeared in the hybrid list in 2018 as fresh replacements in the latest group over the past eight years.

As mentioned above, the top 47 universities in 2010 were in seven countries: US, Australia, Canada, France, Germany, Switzerland and UK (Table 2). Generally, US universities dominated the top 47 universities that met the criteria of being in the top 100 in 2010 according to ARWU, QS and THEs. The US universities comprised more than two-thirds, with 37 universities, followed by UK (14.9%), while the rest had less than 10%. Similarly, the US still dominated the top 57 universities in 2018 with 30 (42%) universities (Table 3), followed by the UK (14%) and Australia (9%), with 8 and 5 universities, respectively. Other countries with top 57 universities included Canada (3), Germany (3), China (2), Japan (2), Singapore (2), Belgium (1) and Switzerland (1). Notably, three universities were dislodged from the top 57 in 2018 compared to 2010, two in the US (Brown University and University of Minnesota) and one in France (Ecole Normale Supérieure).

Table 4 shows the most improved 13 universities in 2018 that did not appear in the top 100 in 2010. The US showed the most improvement, with five universities (38.5%), followed by China, Singapore and Australia at 15.4% and finally Belgium and Germany, each with a single university (7.7%). US universities have dominated the top positions globally, hence this study focuses on two Asian and European countries that have had more improvement lately in terms of global ranking. Consequently, 13 NFWCUs in the US, Australia, China, Singapore, Germany and Belgium are the main focus of the study.


**Table 2.** The combined list of commonly included the top 100 universities in 2010.

Note. \* The universities appeared in the group of NFWCUs in 2010 but disappeared in 2018. Source: Data retrieved on 10–30 September 2018 from THEs (www.timeshighereducation.com/), QS (www.topuniversities.com) and ARWU (www.arwu.org).


**Table 3.** Combined list of commonly-included top 100 universities in the year 2018.


**Table 4.** Newly-formed world-class universities of 2018 from the selected three rankings of ARWU, QS and THE.

Source: Data retrieved on September 10–30, 2018 from THEs (www.timeshighereducation.com/), QS (www. topuniversities.com) and ARWU (www.arwu.org).

Identifying clusters and comparing the findings of the ranking systems for 2018, Monash University, Tsinghua University and Nanyang Technological University have shown significant improvement in performance in recent years compared with their previous rankings in 2010. The trend indicates that the Ecole Normale Supérieure, Brown University and the University of Minnesota have shown rapid fluctuation in ranking for the past eight years. Despite the significant improvement of Tsinghua University and Nanyang Technological University, the progress does not reflect a substantial achievement in the ARWU ranking. As indicated in Table 5, Peking University ranked in the top 100 by both THE and QS in all years and had slight progress in ARWU ranking in the past three years. Brown University was included in all versions for six consecutive years, and then steadily dropped from the family of NFWCUs in 2017 and 2018 (101th to 150th). Due to the rapid decline, it was not included in the ARWU ranking for the past two years. Peking University, Tsinghua University and Nanyang Technological University have consistently continued their performance and variation in the rankings, hence transformed to NFWCUs after slightly increasing in ARWU rankings for the last three years. In all versions (2010–2018), the one ranking of QS has had all NFWCUs in the last eight years, reflecting an improvement compared with the other global rankings of ARWU and THE.

The results indicate that among the 5 hypothesized factors affecting THE ranking in the last 8 years included in the model, only 4 were found to have significant influence on THE ranking in 2018. These include teaching (−0.741 \*), research (−0.999 \*\*), citations (−0.807 \*) and international outlook (−0.293 \*) in which all factors are negatively associated with THE rankings (Table 6). The reason for the 2010 is in relation to the substantial progress in the formation of the NFWCUs in the last eight years. All the variable has negative effects on rankings in 2010 except research. All the variables affecting the ranking explain 45% and 95% variation in the model in 2010 and 2018, respectively.


**Table 5.** Hybrid list of sustainability and fluctuated universities from 2010 to 2018.

Source: Data retrieved on 10–30 September 2018 from THEs (www.timeshighereducation.com/), QS (www. topuniversities.com) and ARWU (www.arwu.org).

**Table 6.** Regression coefficients for factors affecting THE ranking.


\*, \*\* Significant at *p* ≤ 0.05 & *p* ≤ 0.001, respectively, THE = Times Higher Education.

Thus, the results of regression coefficient in ARWU, indicate that among the 6 hypothesized factors affecting ARWU ranking in the last 8 years included in the model, only 2 were found to have significant influence on ARWU ranking in 2018. These include Alumni (−0.845 \*) and PUB (−0.922 \*), in which all factors are negatively associated with ARWU rankings Table 7. All the variables had negative effects on university rankings in 2010 and 2018. The independent variables account for 83% in 2010 and 96% in 2018 variation in ARWU ranking.

**Table 7.** Regression coefficients for factors affecting ARWU ranking.


\* Significant at *p* ≤ 0.05, HiCi = Researchers cited by Thomson Scientific, N&S = Articles published in the Nature and Sciences, PUB = Articles indexed in the SCI and SSCI, PCP= Faculty average score, ARWU = Academic World of World-University.

The results of regression coefficient in QS ranking, indicate that among the 6 hypothesized factors affecting QS ranking in the last 8 years included in the model, 4 factors were found to have a significant influence on QS ranking in 2018. These includes employer reputation (−0.561 \*), student-to-staff ratio (−0.357 \*\*\*), citation per faculty (−0.565 \*\*) and international faculty ratio (−0.294 \*\*) as reveals in Table 8. It was also observed that two variables that are academic reputation (−2.535 \*\*) and the student-to-staff ratio (−0.755 \*\*), had negative effects on university rankings in 2010. The independent variables accounted for 95% and 99% in 2010 and 2018, respectively on the variation on QS ranking.


**Table 8.** Regression coefficients for factors affecting QS ranking.

\*, \*\*, \*\*\* Significant at *p* ≤ 0.05, *p* ≤ 0.01 & *p* ≤ 0.01, respectively, QS = *Quacquarelli Symonds*.

The most current challenge for young universities trying to achieve distinguished global recognition is their internationalization. Based on regression analysis on THE several universities are ranked top in 2018 however their international profile which includes factors such as international students and faculty member is still low compared with top US and UK universities.

#### *5.2. Shared Elements for Sustainability of NFWCUs*

#### 5.2.1. Institutional Practices and Sustainability of World-Class Universities

Around the world, there is great interest on the part of governments in the capacity and performance of elite research universities within national higher education and global ranking systems [26]. In some ways, the level of interest and initiatives varies and in many countries the motives vary and the measures are different [42,58]. Some countries (notably England and the United States) have a widespread influence due to well-established elite HEIs and research institutes founded several years ago and recently ranked at the top level. Other countries, such as Germany, China and Japan, are focusing on promoting some of their existing universities to become WCU. Some are motivating and appraising the global status of their leading national institutions, while Vietnam, India and Malaysia are focusing on building and designing new institutions at the highest national level to gain a global reputation. Mostly second-world economies such as Canada, Australia, New Zealand and South Africa are seeking to break out from national policies and frameworks of funding in an effort to rise. Reflecting changes in recent years, some efforts have been implemented by both developed and developing countries.

Governments and institutions throughout the world have significantly implemented various policies, huge projects and substantial initiatives to translate the governance, function, missions and performance improvement of universities for global rankings [48]. The literature holds that to achieve success as WCU, the quality of teaching and research needs to be given room to innovate through faculty, programs, curricula and enrollment. Considering several important trends today, US-based research accounts for two-thirds (65%), while Asian countries account for around 8.5%. Research capacity has successfully strengthened HEIs in countries such as China, Taiwan, India, Korea and Singapore. Oxford and Cambridge Universities in the UK and Harvard University in the US demonstrate a long history of superior performance. For young universities to transform to WCU may take a decade if most focus on excellence in teaching, research, publications and internationalization. By contrast, the size of

the institution can affect its practices, as most of the early formed WCU are large (≥12,000 students) to extra-large (>30,000 students) in size. It has been reported that the lack of a comparative level of investment and positive initiatives poses a severe threat to institutional practices. Considering Martin Trow's theory on the development stages of higher education, the gross enrollment ratio (GER) is <15% for elite education, 15–50% for mass education and >50% for universal education. The NFWCUs ushered in an era of mass education, as the number of students increased, hence the GER is more than 26.5%. Singapore NFWCUs stand out as an example of a country's that uses internationalization strategy to drive its higher education sector. Thus, by embedding it into core institution missions, expanding participations and align curriculum and institution in order to attract highly talented faculty and students, produce globally impactful research and increase global competitiveness.

#### 5.2.2. World-Class Universities and Government Initiatives

In the ongoing development of newly formed world-class universities (NFWCUs) in most of the desirable countries around the world, as competition for status, several initiatives are implemented to enhance and pursue academic excellence and promote national development. One such effort is huge budgetary allocation of government funding, often in the form of HEI loans and scholarships, projects, mechanisms and reforms [28]. Further, against the background of global competition in science and technology, the pace for governments to develop world-class universities is accelerating. Similarly, world-class universities have undergone fast expansion of development. Governments have supported this growth by introducing reforms in HEIs, such as in governance, taking examples from developed countries such as China. In 2015, the Chinese government released the Developing World-Class Universities and First-Class Disciplines project, known as World-Class 2.0. In 2017, the Ministry of Education announced a list of colleges and universities that replaced Projects 211 and 985, aiming to become a global higher education center.

The literature holds that during the past two decades, previous projects created significant research capacity and contributed to improvements of universities in global ranking systems. With much success, China reached 136 schools in the top 1250 of the US News 2018 rankings, second only to the US, with 221 schools. China currently is also second to the US in some papers and citations in science, although governance and new initiatives are more pragmatic. With the new strategy, the initiatives put great emphasis on top-notch talent education. In short, some studies have found that several countries continue to open up their educational opportunities to other parts of the world.

Recently, some measures have been implemented to improve the quality of NFWCUs. In this context, European and Asian countries has made substantial excellence initiatives and investments in the quality of its higher education institutions. Recently, the Chinese government launched the Belt and Road (B&R) initiatives, which focus on international trade among 65 countries spanned by a common road. The statistics indicate that the B&R project contributes significantly to an increase in the number of international students in the seven countries along the B&R route. Statistics show that in 2003 to 2004, about 25,000 students from B&R counties studied in China. At the end of 2016, the number increased to 200,000, with an average of 22.0% increase per year from 2003 to 2014 from seven top countries: Kazakhstan, Indonesia, Russia, India, Pakistan, Vietnam and Thailand.

Public universities dominate a wide range of performance in the global ranking systems. The federal government in the US has become more comprehensive in a wide range of research and international recognition toward achieving world-class status. Government initiatives and substantial funding affect some of the rankings in American universities and fluctuate compared with those with high-quality support for research, teaching and educational services. Washington University in St. Louis, Missouri, has shown unstable performance, as indicated in Figure 3. The ranking has reshaped the practices of US universities and competition for positions in the ranking has intensified along with competition for resources, especially in the decentralized US system.

**Figure 3.** Rankings of Washington University in St. Louis from 2010 to 2018. Source: Data retrieved on 10–30 September 2018 from THEs (www.timeshighereducation.com/), QS (www.topuniversities.com) and ARWU (www.arwu.org).

The results of variation and disparities of university rankings from 2010 to 2018 where by Singapore, China and Belgium have shown a sharp increase and most progress in numbers of NFWCUs in the global list of leading universities as presented in (Table 9). Singapore, China and Belgium stand out as the most improved countries. The main losers are the USA, UK and France with significantly declining in numbers. In ARWU ranking USA and France, lost 7 and 3 universities while compared to QS ranking UK lose 3 and France 2. In the same case, UK and France dropped in THEWUR by 10 and 3 universities respectively. To large extent, it is inevitable that the progress in some countries forces the exit of universities from other countries. It appears that Switzerland, Australia and Germany sustained NFWCUs with high level of funding and excellence initiative.


**Table 9.** Sustainability per country and evolution in ranking systems between 2010 and 2018.

THEs (www.timeshighereducation.com/), QS (www.topuniversities.com) and ARWU (www.arwu.org).

#### 5.2.3. Sustainable and Catching-Up Countries

In response to US universities, the success factor for global competition is differentiating the mission between universities. The variety of catching-up countries include Germany, France and Japan. These countries are likely to have disparities in global university rankings (as illustrated in Figure 4) because of the languages they use. In the Japan case, half of the drop is linked to financial crisis and which have prevented additional funding expected for making significant progress on the internationalisation activities [26].

**Figure 4.** The variation and total number of Newly Formed World Class Universities (NFWCUs) from 2010 to 2018.

Compared with US and UK publications, most of the research published in languages other than English are not counted in the ranking system. Indeed, the French university system differs from the German education system. France has a well-developed research and technology base in their grandes ecoles. French universities are mostly teaching-focused and therefore different from German universities, which combine research and teaching. The majority of its students are recruited directly from French preparatory schools. There is a strong division of labor between universities and grandes ecoles and between research institutes and the higher education sector. Germany adheres to the philosophy and historical idea that all universities and research universities are equal, which mostly emphasizes global competition. Hence, with excellence initiatives, the government abolished a similar university philosophy.

Currently, the Ecole Normale Supérieure is doing better in ranking because France has a long-established system and the government has attempted to merge teaching-focused universities with research institutes. This has shown strong achievement and high status of French universities in global ranking systems.

Japanese university practices and initiatives for NFWCUs and the capacity of their universities feature a wide range of positive outcomes. Although Asian universities are latecomers in the quest for WCU, global ranking demonstrates how fast they are catching up. According to THE, nine of the top 10 universities in Asia ranked among the top 100 universities in the world. THE has a slightly wider range of countries, showing that four of them were among the top 50 in the world. Based on rapid changes, Japan established its WCU status during the 1990s with Research University 11 (RU11). In August 2011, RU11 launched its mission of getting Japan's research universities ready to face fierce global competition. The government and universities in Japan have made efforts such as the Global 30 Project, a new grand design of the higher education system in which universities collaborate with each other as the official policy of fostering WCU in the first decade of the 21st century. To pursue the goal of being comprehensive and maintaining the breadth of capabilities, universities have engaged in dynamic resources and revitalizing partnerships.

#### 5.2.4. Change in System in All Rankings

Table 6 indicates the variations and disparities of countries and university rankings from 2010 and 2018. The results show changes in a system wherein several universities replaced elite universities in ranking. The successful universities are in countries such as China, Singapore and Belgium. Over the past eight years, these countries have stimulated their domestic universities to achieve world-class status. The findings show that to date, most universities in Asia are increasingly experiencing immense pressure to compete internationally. The growing interest in the sustainability of universities in the global league has become the norm. Tables 4 and 9 also shows further details of the system change in 2010 and 2018. As revealed by the three international ranking systems, the US and the UK showed falling rankings by ARWU (7) and THE (10) and ARWU (2), QS (3) and THE (2), respectively. Similarly, France fell in both rankings and Germany showed a relatively advantageous position in THE (7). China, Singapore and Belgium had leading status and demonstrated a fast increase in the emergence in a change of system and global ranking exercises. This also serves as evidence that there have been several improvements in global ranking and many NFWCUs are likely to improve further. In Singapore, two universities scored slightly better. This reveals that they are yet to be among the top 100 world-class universities. The NTU and NUS presents a model and marked the beginning of new chapter as Singapore's global universities.

#### **6. Discussion of the Findings**

Considering the determinants of success and achievements for NFWCUs in the last eight years, the following are scientifically based frameworks for sustainability and potential complementary critical indicators leading to universities being among the top 100 global rankings by the three selected systems, ARWU, QS and THE.

*Comprehensive and high concentration of talent.* In an attempt to propose manageable lessons learned from the results of top NFWCUs from 2010 to 2018, it is found that the dominant feature of all leading universities ranked at the top is the essential attribute of faculty and students (Tables 6–8). The superior results of NFWCUs and changes in a number of countries are highly sought by graduates, leading-edge research and technology transfer. It is widely acknowledged that NFWCUs are distinguished by the presence of outstanding faculty, a critical mass of academically qualified students and the most researchers and qualified professors. The concentration of talent is the key outlook and sustainability indicator for NFWCUs.

It is clear that the foremost determinant of excellence in top-ranked universities is the presence of a critical mass of top students and outstanding faculty Table 8. Top-ranked universities can select and attract the most qualified students, professors and researchers not only from their own country but also internationally. Indeed, it must be pointed out that the concentration of talent is the essential factor that represents the ability and privilege of NFWCUs to select the best students and staff. Principally, this can be observed at elite universities such as Harvard University and the Massachusetts Institute of Technology (MIT), which are among the most selective universities in the US and Tsinghua University, China's top institution, which admits the 50 best students from each province every year. Faculty quality is considered as one of the most important components in defining academic excellence and has an enormous impact on the quality of the program offered. Ascertaining this important condition is extremely important and a good example can be seen in the champions of NFWCUs such as the National University of Singapore (NUS) and Tsinghua University. According to Salmi Jamil, the proportion of international students at Stanford University is 21% and at Harvard University is 19%, hence the proportion of international faculty and sizable foreign academic staff. The proportion of international academic staff at Cambridge University is approximately 36%.

Indeed, this factor has always been seen in both early formed and NFWCUs such as Oxford University in the UK, Tsinghua University in China and the National University of Singapore. For instance, in the United States, the most selective universities, such as Yale University, measure SAT scores in their recruitment of undergraduate students. As shown in (Table 8), enrollment at Stanford University (US) was 17,354; University of Cambridge (UK), 18,977; KU Leuven (Belgium), 56,351; University of Tokyo (Japan), 28,253; and Tsinghua University (China) 47,762. The huge enrollment size is undoubtedly a major factor in the growth rate gap in the top global ranking. At Stanford and Cambridge Universities, overall enrollment is fewer than 30,000 students. In line with global rankings

and their desire for a good reputation, NFWCUs are also favored by the strong international dimension of foreign academic staff and students.

*Sound financial return base and resources*. The lesson learned from the National University of Singapore (NUS) is the university's success in the top ranking of NFWCUs for 2018 in response to substantial endowment funding. Comparing to elite universities such as Oxford and Cambridge, NUS has a steady source of financing through substantial fundraising. Also, the NFWCUs of Singapore managed to use about 775 million to build up an effective, sizeable portfolio. Comparatively, the US has always managed to cover the top 100 world-class universities in all rankings due to large endowments of about \$40,000 every year.

*Research-based rankings*. Rankings of NFWCUs are also based on government research funding. Therefore, the level of expenditure in one way affects the rankings and respective catch-up countries. Universities in the UK and Switzerland are the best competitors with the highest variations due to abundant resources and good funding.

As illustrated in (Table 5), despite being ranked by all three systems in 2010, the Ecole Normale Supérieure dropped from the ranking despite France's status of having the strongest economy in the world. This is in line with German universities, few of which are placed in the top 100, despite its strong economy in the world.

The discussion and the lesson learned regarding Figure 4 is that China and Singapore have shown the most improvement in the ranking system in the last eight years (2010–2018). Considering the NFWCUs of 2018, Tsinghua, National University of Singapore, Monash University and Rice University showed relative progress toward being among the family of world-class universities for a short time compared to previous rankings. When evaluating against changes in scores in both top 100 university rankings, the following discussion offers a perspective for the variation among countries.

*Inducing excellence-initiatives*: Following the same ideology in constructing world-class universities and enhancing global sustainability and competitiveness, China has been undertaking programs such as the 211 Project and 985 Scheme, which were carried out earlier, compared to other countries such as India. The objectives were measurable and the construction results were obvious. As far as the advantages of cooperation and innovation are concerned, China still has an opportunity to catch up with the UK and the US. The following sustainable strategies for globalization should be used to retain its leading position in the world: modernize the higher education system, attract active research faculty, encourage dissemination of research output, nurture competitive graduate departments and research groups by inviting capable research scholars to join them and integrate resources with full advantages for efficient implementation [39,40]. Thus, Peking University and Tsinghua University are demonstrating better global rankings because they started internationalization earlier than other Asian universities [39]. World-Class 2.0 was released in 2015 with a target goal of developing a number of world-class universities and first-class disciplines by 2020, as well as having higher education and disciplines among the best in the world by 2030 and leading in the number and capacity of WCU and disciplines among the world's best, hence becoming a higher-education powerhouse by 2050. Tsinghua University aims to be a WCU by 2020 and one of the world's best universities by 2050, with greater precision as part of the Chinese dream.

*Developing global students:* Meanwhile, in China since the 1990s, the mass movement of overseas students from many developed countries has increased, driven by both financial and marketing efforts in countries such as US, UK, Canada and Australia [59]. Internationalization of higher education enables students to exchange and flow in all directions, not only from west to east but from less developed to more developed countries. China is now hosting 7% of the 3.3 million international students worldwide, just behind the US, the UK and France [4,60]. In previous years, the data show that China was the leading nation for sending students to study abroad in those developed countries and the top ranked universities. According to 2017 statistics, China has become the nation that is attracting and receiving more international students from different countries of the world [61–63]. As of the end of 2017, China was the most popular destination for international students in Asia. Ultimately, nearly two-thirds of all foreign students in the country (65%) come from markets targeted by China's One Belt, One Road initiative. The literature shows that the top 10 source countries were South Korea, Thailand, Pakistan, US, India, Russia, Japan, Indonesia, Kazakhstan and Laos.

China currently has a significant number of international students seeking a degree, increasing from 184,799 to 209,966 in 2016, while those seeking PhDs increased from 14,367 to 18,051 (www.csis.org). *Programming, curriculum and environment:* The Singaporean government has strategically identified the leading global universities and invited them to set up branch campuses in the city-state, actively promoting collaborations between world-renowned academics and local scholars [64]. According to the policy target, the ministry was well placed to gain an estimated \$2.2 trillion of the world education market. Indeed, the Singaporean government has been successful in convincing world-renowned institutions to establish overseas campuses or offer programs in collaboration with local institutions, thus helping Singapore's universities become globally competitive [61,65,66]. At the same time, the sustainability of the highly ranked top university in Asian countries was based primarily on broader goals, including:


Identifying these critical factors is associated with the sustainability of higher education institutions. According to Shin [67], in their effort to develop NFWCUs, policy makers and institutional leaders pay attention to research productivity, research funding and international faculty and students but other factors such as international alliance, language, geographic location, international climate on campus and economic development are also important for building WCUs.

#### *Experience from Singapore's Newly Formed World-Class Universities*

Building NFWCUs is a national strategy implemented and guided by top policy makers. As early as the 1990s, the Singaporean government was proposing the idea of building a world-class university from its elite universities. In its overall development of higher education, Singapore has had a remarkable achievement in the goal of being among the top research universities. This is particularly true when analyzing the milestone at two of the country's oldest universities, the National University of Singapore (NUS; 1905) and Nanyang Technological University (NTU; 1981). In term of rankings, Table 4 demonstrates that NUS moved up from 101 to 150 on the ARWU, 31st on the QS and 34th on THE in 2009–2010 and to 85th on the ARWU, 15th on the QS and 22nd on THE in 2017–2018, very impressive gains among the top 100 in all three ranking systems. The NUS is free to bring top researchers, including foreign faculty. This involves professors from all over the world, who are paid based on the global market and provided with performance incentives to stimulate competition.

Comprehensive internationalization: The NTU and NUS successes are reflective of Singapore's broader agenda of building Singapore as and education and knowledge hub "global war of talent". Thus, being primarily rely on core institution missions, programs, curriculum, and the large number of international faculty and students. NUS is around 60% foreign faculty, while NTU is around 70%.The overall QS data reveal that NTU and NUS have high faculty/student scores at 95% and 91.8%, respectively and are more advanced in international faculty (www.topuniversities.com).

According to studies by many researchers [66], global ranking is arbitrary depending upon which criteria are applied. However, [67] showed how much ranking could change by applying different measures such as the collaboration index, the total or per capita measure of publications and citations. As many academics have said, a world-class university is recognized by global ranking. The best

conceptual approach to define WCU is to identify the characteristics of the newly formed WCU. Global university ranking systems are currently a measure of world-class universities and there are many theoretical and methodological issues involved in the ranking process [66].

In this context [11,68,69], there are four models for NFWCUs: (1) Upgrade the institution's existing strategic alternative. Several studies found that China has followed this since the early 1980s. Upgrading existing strategies is less expensive but it is a challenge to reform and transform. (2) Merge institutions' existing strategic alternatives. Several studies found that advanced mergers among institutions could be applied to transform a university into a world-class one. These studies found that France, Denmark and China adopted this strategic alternative recently [70,71].

In France, at institutions such as grandes ecoles, the government is investigating the possibility of merging as a regional foundation. It was found in Reference [72] that the Danish government has encouraged setting up world-class universities through grants and innovation funds. In China, some mergers have taken place; for example, Beijing Medical University was combined with Beijing University in 2000 and Zhejiang University was the outcome of merging five different universities. (3) Create new institution strategic alternatives. In Reference [73], it was found that it is most costly to create WCU from scratch. This strategy can be done faster and more successfully than upgrading [42]. (4) Use collaborative strategic alternatives. In Reference [27], a mixed option of upgrading and merging existing universities was noted. This can be done at both the national and international level at the same time. This measurable strategy has recently been implemented in India and Japan.

Given the increased importance of science-based innovation in new technologies, university–private sector partnerships are expected to assume a prominent economic role in national development, as they did in Singapore. Centering on NUS, Singapore established the national science and technology system, made up of diverse public–private research networks, which has contributed to a shift toward a knowledge-based economy. Currently, NUS accounts for about 52% of scientific publications in Singapore and contributes 50% to 70% of the total research and development output.

Going beyond ranking systems, there are many opportunities and connections for the future. Considering the increased interest in global rankings, young learners gain an opportunity to think deeply and work collaboratively across cultural boundaries and differences. With the distinctive roles and missions of world-class universities, young learners are encouraged to take advantage and reassess the value of education. Hence, it is important to find pathways to become globally competitive and adopt local challenges to address employability, social mobility and a high-quality graduate environment. Apart from ranking, looking at the status of any university demonstrates the importance of adding learning opportunities, providing new insights to deepen the collective understanding of the dynamic world of global higher education and research.

#### **7. Conclusions and Sustainable Policy Implications**

This study employed an understanding of sustainability indicators of NFWCUs from 2010 to 2018. This section offers our conclusion of the findings.

Given the recent literature, sustainability for NFWCUs has recently dominated public discourse, mainly regarding their catching up and development. This study investigated the variations and contributions of NFWCUs among countries. It should be noted that NFWCUs represent a concept that manifests in university strength, performance and standards. Global ranking systems are necessary indicators for measuring strengths, sustainability and standards. The study considers the universities included in the top 100 from 2010 to 2018 in three rankings of different methodologies and characteristics. To understand the status and variation among countries, integrating the three rankings helps to minimize the impact of the methodology used, sustainable changes and position gain over time. On the basis of top-ranked world-class universities, it should be recognized that in the last eight years since the THE and QS split into two independent programs, the commonly ranked top 100 universities in 2018 by ARWU, QS and THE were denoted by the top 57, compared with 2010, which was 47 universities. The paper synthesizes the difference between 2010 and 2018, showing that

44 universities maintained their position in all ranking systems among the top 100 without deviation and weight discrepancies and decline of ranking.

Three lower-ranked NFWCUs in the hybrid list of 2010 did not appear in the top 100 universities in 2018, which are covered by some catch-up universities. The Ecole Normale Supérieure, Brown University and the University of Minnesota dropped out of the NFWCU list in 2018, as their rankings fell from the top 100 in THE and ARWU.On the basis of the result of analysis, even though all the three ranking systems have different hypothesized factors, all the variables affecting rankings variations is more observed for 2018 study period.

Likewise, due to outstanding progress made by Tsinghua University, Nanyang Technological University and Peking University, the rankings of these NFWCUs have shown a tremendous sharp increase and homogeneity. These three universities indicate that variations in NFWCUs still have a long way to go to catch up with other world-class universities. A direct implication of the regression analysis also serves as a reminder that to achieve a standard of academic excellence, a university has a chance to become world-class if it continues to increase the quality and significance of research and is highly internationalized with a wide range of subject coverage. Further analysis suggested that in the future, elite universities can lose their ranking if they have no high-level, creative, talent turning out high-quality original research results and making scientific and technological progress.

This study brings the following variable results and conclusions. By systematic comparison, the US and UK dominated the variation of NFWCUs in 2010 and 2018 consecutively.

Climbing up the hierarchy of NFWCUs and since the start of global ranking more than a half-decade ago, elite universities have consistently ranked in the top 100 in all three ranking systems. Monash University represents a particular case and Tsinghua University, National University of Singapore and Peking University have shown overall improvement in their global reputation. Indeed, elements for success at those universities include a regulatory framework governing public universities, structural policies and flexible pathways that enable fluidity of student movements and structural decisions and exceptional innovations such as ongoing government excellence initiatives in Asia and Europe.

Moreover, a more analytical perspective in NFWCUs is that Harvard, Oxford and Cambridge Universities continue to battle year by year for their positions in the ranking systems and these are highly internationalized, comprehensive universities with a wide range of subject coverage. Although most NFWCUs are large in size, academic research quality is a very important element of success. Also, except for private universities, most NFWCUs are relatively large, with an average number of students ranging from 22,000 to 35,000 and correlated faculty ranging from 2400 to 3500. Likewise, most NFWCUs have sufficient technical and administrative personnel to support teaching and research as a critical path to bring the institution to international prominence. Despite the strong constraints of NFWCUs, what distinguishes the top 100 universities in the last eight years from the rest is the enormous amount of available funding. Although some of the top 100 universities are hundreds of years old, they have a concentration of talented academics and students, significant budgets and strategic visions, which are effective approaches to achieve high ranking.

The results are relevant for the strategic planning of universities to improve their reputation. Following empirical data and literature reviews, this paper theorizes that there is a need for this type of study to be undertaken in order to inform society on the gaps in ranking and for NFWCUs to catch up across wide global variation. This involves excellence initiatives, WCU projects, government funding, reforms, policies and strategies to find the correct measurable approaches and key factors to upgrade and transform HEIs into WCU. This study could serve to stimulate new plans and actions toward striving for adequate provision of WCU development. The data could potentially bring forward evidence on institutional practices and sustainable approaches that work well in transforming to WCU status.

Ultimately, for many years, universities in the United States and Europe have been the dominant and best of the WCU, particularly NFWCUs in 2010 and 2018. Despite the remarkable rankings of US and UK, China and Singapore appear to have demonstrated their advantageous positions in the NFWCUs rankings. In the academic year 2017–2018, two universities in China and Singapore were ranked in the ARWU, QS and THEs. The study brings to light the increasing emphasis on the effectiveness and efficiency of government-supported research for more approaches to internationalized initiatives. Hence university expansion is one of the excellent approaches to improve performance in the global ranking systems.

In terms of future implications, our study has endeavored to identify some general patterns in performance among the arguably more research-oriented universities based on the criteria used by ARWU, QS and THE. To improve NFWCUs interpretation factors, it might be better to judge global competitive order and shared indicators of global top best universities by integration of dominant rankings methodology. However, it is debatable whether US and UK universities dominate all global ranking systems. A closer analysis shows that US universities vary and, in some instances, have lost status in the middle and lower end of the ranking systems over recent years and have been replaced by universities from catching-up countries. Our analysis draws attention to how university rankings have generated global variation and sustainability of institutional reputation and the relationship between status and the emerging proliferation of ranking systems. Our paper offers critical contributions to the current literature on success strategies for NFWCUs and their implications for future universities and their quality, not only to create a landscape of new possibilities for reputation but also to reshape sustainability and institutional behavior in the pursuit of enhanced performance. Therefore, considering the sustainability indicators of NFWCUs, many universities benefit from these rankings as an indication of superior output, thus educational and research progress. Universities need to increase their knowledge generation and knowledge transfer and enlarge their focus on research. Providing a more international learning environment, exchange students and researchers with other leading universities in the world and internationalization of NFWCUs policy ant national and university level. In many instances, rankings help to maintain and build institutional position and reputation; rankings are used by students to shortlist their university choices and by universities, stakeholders and policy makers for direct decision-making in higher education, supporting the predominant and leading disciplines and increasing and improving the number and level of foreign teachers, students and international cooperation.

Finally, although the quest for sustainability of NFWCUs remains a challenge around the world, Singaporean and Chinese universities should be applauded for their efforts. Along with various reforms and improvements, in mainland China and Singapore the governments have concentrated their resources on helping their universities become global competitors. Moreover, the Chinese government has focused its grants on a limited number of universities to transform the country's elite universities into world-class universities. In particular, according to policy targets, the Singaporean government has set up campuses for leading universities and has actively promoted active collaboration between world-renowned and local scholars to establish overseas campuses. The empirical results confirm that elite universities are continually leading competitors due to their distinguishing factors: a comprehensive internationalization strategies and activities which driven by; a stable concentration of talented academics and students, a significant budget, research productivity, global and national partnerships and outstanding educational programs. We therefore suggest that it is necessary to take effective major action for NFWCUs in higher education such as maintaining the stability of investments and research productivity, clear mechanism for adjusting the allocation of funds among universities and implementing related strategic initiatives to enhance their international reputation to cope with leading global universities in the future.

**Author Contributions:** This study was conducted under cooperation between all authors. All Authors had contributed to the intellectual content of this entire research study; L.Z. designated the ideas of this study, supervised the overall directions and workflow of the study and findings, reviewed the final analysis, funding acquisition; G.J.M. conceptualized the study, literature review, drafted methodology, collected and analyzed the data and preparation of the final manuscript and C.M.A. participated in gathered the document database from

ARWU, QS and THE, formal analysis, and discussed the final results, and all the authors have read, discussed and approved the final version of the manuscript.

**Funding:** The work describe in this paper was funded by the Jiangsu Social Science Fund for the financed research project, on the paths of governance for "Newly formed" world-class Universities (Project ID number: 17JYA003).

**Acknowledgments:** We would like to acknowledge and express our sincere gratitude to the two anonymous reviewers for their constructive comments and suggestions given for the earlier versions that have significantly improved the quality of this manuscript. We also profoundly thanks Chinese Scholarship Council (CSC), International student's office, and academic staff from the College of Public Administration, Nanjing Agricultural University for the overall comprehensive support to complete this study. We further acknowledge editor's overall guidance and support throughout the process of publishing this manuscript.

**Conflicts of Interest:** The authors declare that there were no potential conflicts of interest reported in this research study.

#### **References**


© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **Sustainability Strategies in Portuguese Higher Education Institutions: Commitments and Practices from Internal Insights**

## **Carla Farinha 1,\* , Sandra Caeiro <sup>1</sup> and Ulisses Azeiteiro <sup>2</sup>**


Received: 18 April 2019; Accepted: 1 June 2019; Published: 11 June 2019

**Abstract:** The Copernicus Declaration of 1994, which was understood as a commitment to sustainable development (SD) by top management in higher education, was signed by many universities. This signature worked as an important driver for these institutions to put different dimensions of SD principles into practice. In Portugal, a Southern European country, six of the fourteen universities belonging to the Portuguese University Rectors Council signed the declaration, but no attempt has been made to evaluate how these public universities integrated education for sustainable development at policy and strategy levels. This paper presents the results of a study aimed at identifying to what extent the integration of sustainability in the fourteen universities was achieved, through their own strategic and activity plans and activity and sustainability reports. A detailed content analysis was conducted on these plans and reports within the period from 2005 to 2014 (the time frame of the United Nations Decade of Education for Sustainable Development), to identify the main commitments and practices. Notwithstanding a lack of national integrated strategies or policies related to education for SD, the results show that the movement made progress at the university level, with good examples and initiatives at several universities. This paper highlights the importance of analyzing the content of plans and reports from higher education institutions (HEIs) when intending to assess and define a country profile for the implementation of sustainability in the educational sector. In addition, this research, conducted in Portugal, may be helpful to understand and value how SD is being applied in the policies and strategies of other European HEIs, as well as to share and encourage best practices and ways of improvement.

**Keywords:** commitments; education for sustainable development; Portuguese; practices; sustainability reports; universities

#### **1. Introduction**

For the decade from 2005 to 2014, much research has focused on how sustainable development (SD) was incorporated in universities, especially because higher education institutions (HEIs) signed declarations, charters, and initiatives (DCIs) to demonstrate their top management's commitment to sustainability in their system [1–3].

By the end of the above-mentioned decade, more than 1000 universities had ratified DCIs, so HEIs were engaged in fostering transformative SD [2]. Until now, there is a scarcity of investigation looking at the extent to which planning for SD can help HEIs to assess their performance and to determine whether the aims of their strategies and practices have been met [3].

In Portugal, earlier research showed that embedding sustainability (the "top-down" approach) is insufficiently developed in Portuguese governmental institutions at university level [4,5].

In addition, the debate concerning HEIs' role towards SD has recently begun [6,7] and the few events organized so far were mostly dedicated to the environmental perspective [8]. Moreover, SD policies are key factors for a university's successful engagement concerning sustainability matters and indicate how active they are in this field [8]. One of the levels of sustainability integration in higher education (HE) is at the institution level within the macro HE public policy system [9]. Nonetheless, no attempt has been made to assess how Portuguese public HEIs are integrating education for sustainable development (ESD) at policy and strategy levels, and how the documental analysis of HEI plans, reports, and strategies can be a useful approach to evaluate SD integration in universities. The research question is to what extent ESD has been integrated in the Portuguese public HEIs' policies within the United Nations Decade of Education for Sustainable Development (UN DESD) 2005–2014, and consequently to provide insights about their (best) practices.

The purpose of this study, conducted within the timeframe of the United Nations Decade of Education for Sustainable Development (UN DESD) 2005–2014, is to evaluate the extent to which ESD has been integrated in Portuguese public HEIs through the treatment and analysis of the universities' (i) strategic activity plans (PEs), strategic plans and development plans (PDEs), and activity and operational plans (PAs); (ii) activity reports (RAs), strategic activity reports, sustainability reports (RSs), and annual financial reports (RCs); as well as (iii) responsibility and assessment frameworks (QUARs) (QUAR ("Quadros de avaliação e responsabilização") illuminate the universities' mission, their strategic and operational goals, their key performance indicators and aims, as well as the financial and human resources available to facilitate moving towards targets and the achievement and effectiveness of such targets).

These plans relate to what HEIs are planning to accomplish in the short or medium term, depending if it is an annual or a quinquennial program, and the reports relate to what has been achieved from within the plan or beyond the plan.

#### *1.1. Universities' Commitments to Implement ESD*

In October 1990, the Taillores Declaration was signed by 30 universities worldwide. This early declaration recognized the fundamental role that universities should have in the future concerning the implementation and dissemination of sustainability:

*Universities have a major role in the education, research, policy formation, and information exchange necessary to make these goals possible. Thus, university leaders must initiate and support mobilization of internal and external resources so that their institutions respond to this urgent challenge* [10].

Later, the 1992 Conference of European Rectors at the United Nations Conference on Environment and Development (UNCED), which took place in Rio de Janeiro, made an urgent appeal for the involvement of universities in SD and for an inclusive strategy for building a sustainable future which is equitable for all. In Europe, this declaration was signed by more than 320 HEIs in 38 countries [11].

In 1994, the Copernicus program developed its own strategy on the ten action principles to preserve the environment and promote SD, which was signed by 196 universities [12]. The universities' role was defined as follows:

*It is consequently their [universities] duty to propagate environmental literacy and to promote the practice of environmental ethics in society, in accordance with the principles set out in the Magna Carta of European Universities and subsequent university declarations, and along the lines of the UNCED* [Rio Conference in 1992] *recommendations for environment and development education* [12].

In May 2005, at the European Higher Education Ministerial Conference held in Bergen, Norway, there was a strong reference to SD for the first time. It was said, when describing the Bologna Process, that "our contribution to achieving education for all should be based on the principle of sustainable

development and be in accordance with the ongoing international work on developing guidelines for quality provision of cross-border higher education" [11].

At the United Nations Rio + 20 conference in 2012, the commitment of Higher Education Sustainable Initiatives (HESI) was announced, including teaching sustainable development concepts, encouraging research on SD, making campuses more sustainable, and involving the community in all these actions, committing institutions to concrete results and actions [13].

Additionally, the UNESCO World Conference on ESD, held in Aichi-Nagoya (Japan) in 2014, adopted a declaration and a call for urgent action to further strengthen and scale up ESD, where HEIs have a special role [14], namely in transforming societies and in key aspects of citizenship.

In the post-2015 DESD agenda, these characteristics were emphasized and linked to the establishment and achievement of the sustainable development goals (SDGs) defined by the United Nations in 2015 [15]. In fact, the seventeen SDGs were set placing education at the heart of the promotion of SD [16], proposing a HE field that is greatly influenced by the global sustainability agenda as well as by the management education requirements [17].

From a worldwide survey linked to the seven dimensions of the recognized university system [2], it was concluded that there is a strong relationship between SD commitment, integration, and the signing of DCIs, showing that there are two HEI clusters:

*"the ones at the forefront, which show high commitment, have signed a declaration or belong to a charter, and have engaged in implementing SD; and those HEIs, which are lagging in commitment, implementation, and declaration signing"* [2].

#### *1.2. A Worldwide Integration of ESD in Universities' Strategies and Policies*

HEIs can implement ESD in several dimensions in order to be as holistic as possible. The more common dimensions are: (1) Institutional framework (i.e., the HEIs' commitment); (2) campus operations; (3) education: courses on SD, programs on SD, transdisciplinary curricular reviews, including "educate-the-educators" programs (which promote competencies in EDS to enable an integrated approach of knowledge, procedures, attitudes, and values in teaching through multidisciplinary and transdisciplinary teams [18]); (4) research; (5) outreach and collaboration; (6) SD through on-campus experiences, working groups, policies for students and staff, among other practices; and (7) assessment and reporting [2,19].

Universities worldwide are experiencing an increasing trend towards responding to the need for sustainability and various knowledge gaps [20], as well as collaborating and contributing to the generation of sustainability values, attitudes, and behaviors within future regenerative societies [21]. Regarding some European countries, access to quality education is so critical for development [22] that the European Parliament has continuously called for the allocation of its budget to investment in this sector [23]. Universities can use low-carbon campuses as living laboratories in shaping the leaders of future sustainability thought. Many HEIs are already involved in mainstreaming the environment and sustainability into their curricula, training, research, and community engagement activities [24].

From the results of surveying a sample of universities from Germany, Greece, United Kingdom (UK), United States of America (USA), South Africa, Brazil, and Portugal [8], it was reported that there is a widely-held belief that SD policies are essential for HEIs to successfully engage in matters related to sustainability and that such policies show how active they are in this field. Therefore, a university must be considered active and have formal policies on SD as a pre-condition for successful sustainability efforts [25].

Considering HEIs' degree of commitment to and institutional trust in sustainability in USA, it was noted [25] that universities are uniquely positioned as knowledge disseminators, behavior consolidators, and idea innovators towards a resilient and impartial society, as they offer a superior learning environment and campus lifestyle experience to initiate a more holistic understanding and contemplation around sustainability.

Therefore, HEIs have embedded sustainability initiatives into their core activities, curriculum, research, community, and operational, to respond to the worldwide transformation towards a sustainable future [26].

#### *1.3. An Implementation Research Gap in Portuguese Public Universities*

Despite international studies on ESD in European universities ,which provide best practices and examples [27–29], this area represents a gap in higher education research in some countries (e.g., Czech Republic, Poland, Spain) [30–32] and the insufficient number of studies in Portugal concerning strategic environmental assessment was emphasized [33].

These detailed, national-scale studies can contribute to a better evaluation of HEIs' levels of effort and success in contributing towards encouraging worldwide sustainable development and the role of academia in meeting this purpose [17].

In 2007, which falls within the decade 2005–2014, the Portuguese Government passed the Decree-Law 242/2007, which transposed the Directive 2001/42/EC, promoted the effective institutional autonomy of universities [34], and facilitated environmental assessments regarding the effects of certain plans and programs [32].

In comparison to other European countries, Portugal was far behind in externally-oriented activities aimed at building capacity within local communities to promote SD, and Portuguese HEIs were classified as "laggards" and/or "late majority" in integrating SD in education, in research on sustainability, and in inclusive development in universities, in particular when compared with other Southern European countries [6].

Despite having signed Declarations and/or Charters, Portuguese public HEIs may or may not have implemented SD, while others that did not sign any commitment have engaged in implementing sustainability.

Regardless of previous research, it is important to comprehend how Portuguese public universities are applying ESD at policy and strategy levels (between 2005 and 2014), since no attempt has been made to evaluate their commitments and practices in a systematic and detailed way.

#### **2. Methodology**

#### *2.1. University's Sample of Universities*

Considering the UN DESD 2005–2014, the University Higher Education Institutions (UHEI) sample was based on the effective members of the Portuguese University Rectors Council (CRUP) during the analysis period (2005–2014), which correspond to all public universities. These HEIs comprised: UAc—University of the Azores [35], UMinho—University of Minho [36–38], UAb—Universidade Aberta [39,40], UP—University of Porto [41,42], UAlg—University of Algarve, UTAD—University of Trás os Montes e Alto Douro [43], UÉ—University of Évora [44,45], UBI—University of Beira Interior [46–49], UC—University of Coimbra [50,51], UTL—Technical University of Lisbon, UL—University of Lisbon, ULisboa – Universidade de Lisboa [52], UNL—NOVA University of Lisbon [53], UA—University of Aveiro [54], and UMa—University of Madeira.

In July 2013, two large public universities, UTL and UL merged to increase their scale, attract a larger volume of students, capitalize on the prestige of their faculties, and help them to achieve a greater leadership role in the European context. ULisboa "brings together various areas of knowledge and has a privileged position for facilitating the contemporary evolution of science, technology, arts and humanities [52]"

These public HEIs, together with ISCTE-IUL—University Institute of Lisbon [55] and UCP—Universidade Católica Portuguesa [56], represent the core of the Portuguese national higher education system [57].

The creation in 1979 of CRUP—Portuguese University Rectors Council, a Portuguese university associative structure, constituted a major step in the decentralization of the Ministry of Science, Technology, and Higher Education (MCTES) responsibilities for Higher Education [58]. One of its major working areas is guaranteeing universities' coordination and their representativeness, while ensuring their autonomy [57] (see Appendix A, Figure A1).

Despite the researchers' efforts, it was not possible to obtain supplementary documentation from all the universities that belong to CRUP.

The final UHEI sample turned out to be 14 public universities and some had similar characteristics such as geographical location, number of students, and campus area (see Table 1).

Confidentiality was ensured by allocating an alphanumeric identification to each public university (HEI\_01 to HEI\_14) so that the names of the respective institutions did not appear in the publication findings and results.


**Table 1.** Characteristics of Portuguese public universities.

Legend: Information not available (NA), square meters (m2), carbon dioxide emissions (CO2), Eq. (Equivalent), tonnes (ton); Each year corresponds to academic year; academic year. (a) 2012/2013 (b) 2014/2015; (c) 2015/2016; (d) 2016/2017; (e) 2017/2018; (f) unknown. Source: CRUP, 2018.

#### *2.2. Data Collection and Time Frame*

This study used a qualitative approach [59] and a detailed content analysis method. Institutional documents were analyzed to:


The following types of documents corresponding to the period 2005 to 2014 (i.e., a 10-year period; see Table 2) for each HEI, were:


The data were collected between 1 January 2015 and 30 June 2016, through public university websites, email contacts, and some UHEIs' documentation centers, mainly due to their willingness to participate in this study. After the data collection period, no further documentation was considered despite its availability on websites.

Eventually, universities might publish this type of documentation, but it was not available for the researchers during the time frame of the collection period despite their efforts.

Overall, 168 documents from the 14 public universities were gathered for treatment and analysis.

#### *2.3. Documental Approach of Public Universities' Sustainability Integration*

HEI\_01, HEI\_02, HEI\_03, HEI\_04, HEI\_05, HEI\_06, and HEI\_07 contributed 85% of all the collected documents (see Appendix A, Figure A2). Even though seven universities provided the vast majority of the institutional document sample, the aim was to find out how each public HEI implemented sustainability and their commitment to SD, and to provide insights about best practices.

The year 2011, which was the year in which Portugal came under the international financial assistance program, corresponded to the highest number of documents gathered. This may be explained by the increased need to support financial reports with long-term planning.

Considering the first half of the UN decade 2005–2014, corresponding to the period from 2005 to 2009, concerning document type, PAs, PDEs, PEs, RAs, Strategic Activity Reports, and RSs represented 83% of all the documents.

In the second half of the period 2010–2014 there was not much difference (80%). RS accounted for 10% and 4% of the collected documents in the first and second half of the decade 2005–2014, respectively, and were published either by HEI\_01 or HEI\_03. From the second half of the DESD, around 33% and 43% were RAs/Strategic Activity Reports and PAs/PDEs/PEs, respectively. There seems to have been more activity planning than reporting, which might not be so true if RSs and RCs were combined.

The scenario was quite different when analyzing the documentation obtained in the period 2005 to 2009, as it seems there was more reporting and less planning. Adding RC (5%) and RS (10%) accounted for almost 66% of reporting activity altogether (see Figure 1).

From 2005 to 2014, almost 80% of the collected documentation was related to activity planning or reporting (see Appendix A, Figure A2). Despite the few sustainability reports published by the public HEIs (only two did so, UMinho and UP), they are of utmost importance for the content analysis concerning sustainability implementation because they were published during the UN Decade.



**Figure 1.** Distribution of document type through the UN decade 2005–2014 in Portuguese public universities.

*2.4. Documental Sample Data Treatment and Analysis*

The data treatment and analysis were divided in a four-step approach:

1. When collecting documents, few universities possess documents such as RC and QUAR. Since this is the case, this constitutes a drawback in the study to (better) assess policies and strategies at university level, so it was the first cut in the treatment phase. From an overall sample of 168 documents it was reduced to 139 (the "major documents") (see Table 3). From here, the data treatment was made.


**Table 3.** Four-step approach in data treatment and analysis.

Note: The documents that were treated and analyzed from step 1 onwards neither include QUARs nor RSs.

The documents were selected, taking into account neither type nor university origin, to be treated and analyzed considering the highest frequency of keywords (see Table 4) in the defined coding system obtained in the content analysis of a previous study [4]. The following results were, in descending order, "Integration or intervention or implementation" (the main reference found), followed by "Environmental Education" (these two were the main references), then "University Higher Education or University" and "Sustainability (ies) or sustainable (s)".

2. The content was then analyzed in a systematic review, where a node corresponds to a public UHEI and each subcategory to a type of document. This coding technique was used to analyze the documents. As coding is a process to generate categories, the analysis started by using descriptive coding, where words and sentences from document transcripts were labeled using relevant words or phrases [60].

	- Institutional framework (Dimension #1);
	- Campus operations (Dimension #2);
	- Education (Dimension #3);
	- Research (Dimension #4);
	- Outreach and collaboration (Dimension #5);
	- SD through on-campus experiences (Dimension #6); and
	- Assessment and reporting (Dimension #7).

#### **Table 4.** The highest frequency of keywords.


\* We added this keyword as it was found to be important in many of the documents analyzed.

The themes where ESD has been implemented in HEIs were organized in dimensions and corresponded to subcategories. Each subcategory was called a sustainability implementation action (SIA) within the content analysis methodology [59]. In the end, the coding system was rearranged again based on the number of codified references, and the sustainability implementation actions (SIA) renamed, which were obtained after the treatment and analysis of the major documents.

The process consisted of organizing the disclosed data into distinct categories and/or new nodes, through a classification.

Every time a document was treated and analyzed; the code was modified to reflect the correct adjustments. This was; therefore, a collaborative process based on diversified readings before treating and analyzing the available documentation—139 documents from the 2005 to 2014 period—from which at least three adjustments were made to some of the items (a suggested procedure [61]).

The dimensions of the recognized university system [2] were used, as well as the themes associated with each aspect as a proxy of integration sustainability in each HEI. This was a cataloguing method in which an organized codebook was produced.

Lastly, all data contributed to the definition of a country profile for the implementation of sustainability in the HE sector.

For the qualitative content analysis, NVIVO (version 11) software (QSR International Pty Ltd, Victoria, Australia) was used [62].

#### **3. Results**

#### *3.1. The Sustainability Implementation Actions in Portuguese Public HEIs*

Overall, considering the seven dimensions [2], 66 themes were found as sustainability implementation actions (see Figure 3).

All Portuguese public universities seemed to have been implementing sustainability and more than 50% of actions were not exclusive to a single UHEI (see Table 1). Among the seven dimensions, "campus operations," "outreach and collaboration," and "SD through on-campus experiences" represented almost two thirds of the total sustainability implementation actions (see Table 5 and Figures 2 and 3). It; thus, seems that these were the main dimensions by which the Portuguese UHEIs implemented sustainability through strategies and policies.

**Figure 2.** Number of sustainability implementation actions by each dimension.

**Figure 3.** Number of sustainability implementation actions in each public university in each dimension.




 of Sustainability implementation actions; Legend: Green House Gases

#### *Sustainability* **2019** , *11*, 3227

Considering the number of sustainability implementation actions (see Table 5) throughout the HEIs, the top three were:


Taking into consideration the treated and analyzed documents, universities' actions relating to ESD seemed to have been taken in "isolation" and were not integrated in a whole institution approach. Each HEI acted according to a tank of actions—"think tank" (see Figure 4).

**Figure 4.** Think tank of initiatives in Portuguese public universities.

Each university may have taken one, or more than one, path to integrate their strategies and policies on sustainability, but it seems any integration did not keep up with the simultaneous pace of action.

The findings in Portuguese public HEIs also suggested that identical SD integration during the DESD 2005–2014 could have occurred for different reasons:


#### *3.2. DCIs and the Commitments of the Portuguese Public Universities*

As of 1 February 2018, 502 institutions had signed the Taillores Declaration. However, in 1990, the NOVA University of Lisbon (UNL) was listed as the only Portuguese signatory HEI, according to the Association of University Leaders for a Sustainable Future [10].

The findings indicate that UNL was deeply involved in the outreach and collaboration and institutional framework Dimensions through the following sustainability implementation actions: (1) Joint degrees with other universities, and (2) the existence of policy and a strategic plan for implementing SD in the University.

Besides having signed the Taillores Declaration, UNL belonged to the Copernicus Charter in 1994. According to these documents' principles, sustainability should be incorporated in a university's faculties, departments, and other entities. The signature by UNL of both the Declaration and the Charter signaled an official commitment to SD by this university.

Nevertheless, other Portuguese HEIs also signed the Copernicus Charter, such as UTL, UP, UMinho, UL, and UCP. The results concerning UMinho and UP will be shown in Section 3.3.

The results indicate that like UNL, UTL was involved in the outreach and collaboration Dimension through the creation of joint degrees with other universities.

The overall results indicate that UNL and UTL (which, after the merger with UL, resulted in ULisboa), representing almost 30% of all HEIs' students, were both involved in the creation of a joint degree as mentioned. Nonetheless, it cannot be assured through any DCIs that this fact is due to their commitment to SD.

#### *3.3. Commitment to SD of Universities with Sustainability Reports (RS) and DCI*

UMinho and UP were the only two out of the six Portuguese Copernicus Charter signatories that developed the "assessment and reporting" through sustainability reports. RSs enable organizations to take into consideration the impact of a wide range of sustainability issues, allowing them to be more transparent about the risks and opportunities [63].

Owing to UMinho's strong cultural activity, this HEI uses the Global Report Initiative (GRI) as guidelines for sustainability reporting (2010 and 2011) and improved its methodology in 2012/2013 [36] (pp. 113–114) by including a new (cultural) dimension [37].

According to the RS from 2011 [36] (pp. 113–114), globally UMinho is on its way to sustainability considering economic, environmental, and social indicators, namely due to its direct and indirect impact in the local economy. As an example, the production of dangerous solid waste had been reduced by 2.5 ton from 2009 to 2011 and the 2015 emissions of CO2 equivalent (ton) × 1000 ton. CO2 equivalent were 16 in a campus area of 40 ha (see also Table 1).

Nevertheless, environmental performance should be improved to reinforce UMinho's commitment to sustainability, according to the University Rector (see Table 6). From the analysis of the documents, the sustainability implementation actions of UMinho were mainly based (almost 50% of the total number of UMinho's initiatives) on the "campus operations" Dimension, either through (1) plans to improve energy efficiency; (2) energy efficient equipment; (3) policies and activities to reduce paper consumption; (4) plans to improve the management of waste; or (5) green purchasing from environmentally and socially responsible companies. There were also actions based on "institutional framework" through the existence of policies for implementing SD in the university.

The National Strategy for Ecological Public Purchases by Resolution of the Council of Ministers (i.e., a government decision) was found to be used by UMinho concerning green purchasing as well as the Energetic Efficiency Program in Public Administration (Eco.AP) regarding energy efficiency.

There are some best practices in this university seen in the Institute of Science and Innovation for Bio-Sustainability (IB-S) and Landscape Laboratory.

The first Portuguese HEI that used GRI guidelines was the Engineering Faculty of University of Porto (FEUP) in 2006, and from 2008 onwards; however, the RS are only related to the faculty and not the whole university. The GRI model was used to assess, monitor, and report sustainability with a focus on the academic community, operations, teaching, and impact on society, which seems to have some similarities with the Sustainability Assessment Questionnaire (SAQ).


**Table 6.** Sustainability reports: UP and UMinho.

It should be noted that FEUP is concerned with all Dimensions and not only environmental ones [41].

These sustainability implementations actions by the University of Porto seem to have been based on many different Dimensions. Concerning the "campus operations" Dimension, actions seem to occur through (1) sustainable landscaping; (2) policies and activities to reduce paper consumption, such as e-communications or double-sided copying; (3) renewable energy usage, through the implementation of photoelectric performance systems; and (4) energy-efficient equipment.

There were also actions relating to "SD through on-campus experiences," through (1) policies that promote SD for all students and staff; (2) sustainable practices for students; (3) a SD working group with members from different departments; (4) SD efforts that are visible throughout the campus; and (5) student participation in SD activities, such as collaboration in multiple social solidarity projects.

Concerning the "assessment and reporting" Dimension, UP seemed to have implemented sustainability through (1) RS, and (2) the assessment of SD issues using SD integration instruments and tools within the University through the total management system (SGT); the implementation of consumption monitoring routines (namely, student participation in SD activities through collaboration in multiple social solidarity projects, and the disclosure of RS); and some best practices (namely the optimization of equipment and system schedules through the centralized technical management system (SGTC) and the "paper calculator" software developed by the "Environmental Paper Network" [42] and G.A.S.PORTO - Oporto Social Action Group).

There seems to have been be special care taken regarding the publication of RS by UP/FEUP between 2008 and 2011 and the integration of instruments and tools to assess SD issues.

Regarding the "outreach and collaboration" Dimension, the action related to the involvement of academic staff in voluntary advisory activities in SD seemed to be one of the initiatives.

The UP's "institutional framework" demonstrates a commitment to the inclusion of SD in the vision, mission, goals, and objectives of the University.

The extent to which UMinho and UP were able to integrate sustainability into their strategies or policies can be found through the actions organized in themes. From there, not only did these HEIs seem to have implemented sustainability internally through campus activities and on-campus experiences, but they also did it through outreach and collaboration (external routes). Both HEIs were committed to SD within their institutional framework and deeply involved in the assessment and reporting Dimensions.

#### *3.4. Commitment to SD of Universities without DCIs or RS*

There were universities that had not signed any DCI or published any RS but were committed to SD and implemented sustainability actions.

Many HEIs used the Energetic Efficiency Program in Public Administration (Eco.AP) regarding energy efficiency in the "campus operations" Dimensions (which was the case of HEI\_04, HEI\_05, HEI\_06, and HEI\_08; see Table 5 and Figure 5).

The implementation of "SD through on-campus experiences" was found in many of the studied universities, as well as other sustainability implementation actions, such as policies that promote SD for all students and staff; in these areas, SD efforts were visible throughout the campus and some best practices were found (e.g., "knowledge sharing" and a "cultural training program").

Regarding "outreach and collaboration," the actions found were: (1) SD partnerships with other society stakeholders (HEI\_08 and HEI\_13), and (2) academic staff involved in voluntary advisory activities in SD (e.g., HEI\_08).

One of the universities played a role in the environmental area with the creation of a sustainable campus that resulted from a partnership with GALP Energia (a Portuguese energy company) and others. Another initiative by this university involved the creation of synergies between sports and health, involving a stadium in the promotion of common projects with schools (best practice). Moreover, another university had a role in the promotion of sports and adapted sports, like canoeing, sailing, and

pp p

adapted sailing, as well as in the creation of research centers and/or associated laboratories (hosting researchers from other universities).

Concerning the "education" Dimension, some HEIs created study programs (e.g., Masters— Sustainable Energy, Environment and Sustainability, PhD—Sustainable Energy Systems, which was financed by the Massachusetts Institute of Technology (MIT) program in 2007 [64], Global Change (Climate Change and Sustainable Development Policies), Social Sustainability and Development) in areas such as energy, global change, sustainability, environment and sustainability, social sustainability and development, or a combination of these terms.

In one university, the gathering of professors from different faculties, departments and research and development (R&D) units was a path to promote interdisciplinary collaboration in teaching and development. This leverages talent and financial resources and creates awareness on sustainability issues, namely in the areas of energy and SD.

At one of the studied universities, the commencement of a doctoral program in the academic year 2010/2011, which is an interdepartmental program between two departments, is a good example of a university offering education with a transdisciplinary focus. The sustainability implementation action was evidenced by course syllabuses of courses or programs on SD.

In the "research" Dimension, one university showed the existence of patents in the field of SD.

Towards a country profile for Portugal for the implementation of sustainability in Higher Education, on the basis of their likelihood in the "think tank" (Figure 4), the sustainability implementation actions were classified according to the quartiles (see Figure 5) for the overall number, for the dimensions of campus operations, outreach and collaboration, and SD through on-campus experiences (the top three).

**Figure 5.** Box plot for the top three sustainability implementation actions in Portuguese public universities.


Figure 5 is a box plot. Considering the top three Dimensions, the first, second, and third quartiles overlapped. This means that 75% of the Universities have taken this path to implement one or two sustainability actions. For the Dimensions education and research combined, four actions were taken in 75% of the universities.

Universities seemed to have integrated SD through multiple and simultaneous actions at their own rhythm and pace.

These findings showed no apparent relationship with the number of students or campus area because the results followed all of the steps explained in Section 2.

#### **4. Discussion**

Many European universities have integrated SD into their academic systems. There are also important connections between commitment, integration, and the signing of a DCI [2], relating to the leverage of values, attitudes, and behaviors within present and future regenerative societies [21].

The results presented in this paper show that if a university signs a declaration or a charter it seems to lead to a commitment to SD, no matter how narrow it may be, partly through the implementation of several sustainability actions. This was the case of at least four universities in Portugal (UP, UMinho, UNL, and UTL). However, sustainability implementation was present in all the other studied universities.

During the DESD 2005–2014, the results show that Portuguese public universities implemented sustainability through diverse and multiple actions, mostly by (i) establishing partnerships with other society stakeholders; (ii) implementing policies that promote SD for all students and staff; (iii) signing DCIs within SD, ESD, or sustainability during the UN decade; and also (iv) by promoting best practices.

Aleixo et al. (2018) and Arroyo et al. (2017) [65,66] refer not only to the importance of putting into practice universities' transformative role in SD by including sustainability in an institution's agenda, strategies, and best practices to promote said agenda, but also by the institution remaining engaged in the field despite facing the usual implementation problems, varying from restricted resources to lack of trained staff [3], deficient organizational structure, inertia, and resistance [66].

Based on the evidence of sustainability implementation actions, concrete proof for whether universities were committed to SD, a four-group classification was built to measure how far the policies and strategies were integrated. It showed that despite some universities having done more than others regarding the dimensions [2], all of them were engaged in SD implementation at their own pace. This is in line with published literature about Portuguese HEIs [65] that recommend a further development of sustainability initiatives for several Portuguese universities.

More than 50% of the actions in Portuguese public universities were not exclusive to a single university. Additionally, the "campus operations," "outreach and collaboration," and "SD through on-campus experiences" Dimensions represented about two thirds of the total sustainability implementation actions. Therefore, the way by which ESD has been integrated in Portuguese public universities within the United Nations Decade of Education for Sustainable Development 2005–2014 seems to have been a bottom-up approach. A university must have policies on SD which are in line with [18] when mentioning them as a pre-condition for successful sustainability efforts.

Sustainability reports are a suitable tool for universities concerning SD incorporation, but this is not a common practice [1]. RSs are a tool increasingly used by accreditation bodies, governments, and students [67]. This seems to correspond to the presented findings, as UP and UMinho were the only two universities that produce RSs.

RSs have a large potential for the process of sustainability development integration in HE, namely for organizational change, stakeholder engagement processes in RS, link between RS and general sustainability management, and relationships between existing reporting indicators, tools, and management standards [68]. Thus, the development of RSs at universities in Portugal should be widely encouraged. Aleixo et al. (2018) [66] mention that UMinho is in a SD implementation phase, due to university sustainability reports, and so this university seems to be an early adopter.

From this study's findings, best practices regarding green campus procedures were found in many of the studied universities. Indeed, campus operations are among the more commonly applied ESD domains in universities ([9,66,69]. At this point, it should be said that the data used for this characterization can be underestimated and differences between institutions may be attributed to cataloguing methods, lack of documentation, or a less systematic search where the terms (e.g., "green campus procedures") were not formally stated.

Regarding the "outreach and collaboration" Dimension, namely "partnerships with other civil stakeholders (e.g., Non-Governmental Organizations (NGOs), municipality, regional government, etc.)," many best practices were found in Portuguese public universities (e.g., UBI and UA) which seems to be not quite in line with [70], who reported that Portugal was far behind in externally-oriented activities aimed at building capacity within local communities to promote SD.

Implementation actions relating to the "education" and "research" Dimensions were not intensely found, which is in accordance with [6] that classified Portuguese universities as "laggards" and/or "late majority".

There may be significant advancements in the operational dimensions of a university, in curricular and educational transformation as well as in research and outreach activities [71,72], but in most cases, sustainability has not yet become an integral part of the university system [73].

*Notwithstanding its improvement in recent years, the requested paradigm change from un-sustainability to sustainability in university systems is not yet fully identifiable* [74].

Even so, Portuguese universities show good examples of sustainability interdisciplinary curricula, particularly at the post-graduate level. The breadth and interconnectedness required for implementing the SDGs make it evident that experts from different subjects and sectors must work together to deliver the goals [16], as well as that future research should concentrate on the challenge of measuring and assessing the differing conceptualizations of "sustainability" within what the curricular offers [68,75].

Many universities are already involved in sustainability through the curricula, training, research, and community engagement activities [24]. This difference may be attributed either to the localization of public universities and/or the lack of documentation from some universities.

Communication is a core function of higher education [9]. In terms of ESD coordination and communication at the national level, it should be mentioned that there is an existing gap arising from the lack of ESD at governmental policy and strategy levels either by the Portuguese Government or the Ministry of Public Universities [4,5].

Nevertheless, there has been effective coordination between universities regarding national and international programs like Eco.AP and the MIT 2007 Program.

A detailed and deep content analysis of several documents, namely the strategic and activity plans, showed that, during the UN DESD 2005–2014, Portuguese public universities implemented sustainability actions in many different ways and Dimensions when compared with earlier studies.

Nevertheless, the initiatives found in each university were not integrated within a whole-school approach [19]. A whole-university approach for embedding sustainability in the university is fundamental for a transformation in learning and education for sustainability with interdisciplinary collaboration between academics. This is critical for promoting the needed transformation in students to become agents of a sustainable future [9,25].

Usually in these types of studies, where a profile of a region is drawn, data are gathered only by questionnaire or interview survey [75]. This systematic analysis of gathered documental data was the basis for the characterization of a country profile for Portugal for ESD implementation in universities and allowed a detail analysis usually not possible through surveys, in which response rates are often low.

Based on the searched and identified actions, a "think tank" (a tank of actions) may be widened, and a cooperation network—SharingSustainability4U—established with a list of best practices and areas for sustainability improvement, irrespective of the university's dimensions. Single universities may support and benefit from being a node in a university network for sustainability [76]. Collaboration and support among universities are key success factors as universities have not implemented sustainability at the same pace, to the same extent, and in the same Dimension(s). The Portuguese University Rectors Council can have a key role as mediator or even coordinator of this network.

#### **5. Conclusions, Future Lines of Research, and Limitations of the Study**

#### *5.1. Main Conclusions*

During the United Nations Decade of Education for Sustainable Development 2005–2014, Portuguese public universities integrated sustainability into university policies and strategies mainly through "campus operations," "outreach and collaboration," and "SD through on-campus experiences" Dimensions. Universities implemented sustainability through actions, many of which were not exclusive to only one university. One hundred and thirty-nine documents from fourteen universities were treated and analyzed to provide a better understanding of the progress regarding ESD implementation in Portuguese public universities and to find the main commitments and practices. The step-by-step treatment and systematic analysis of those documents helped to understand and value the possible sustainability implementation actions and university results on the strategies and policies of the public universities.

From this research, some important conclusions may be drawn:


The aims regarding the institutional document analysis from internal insights were accomplished.

#### *5.2. Limitations of the Study and Future Research*

This study had some methodological limitations. For the relevant period (2005–2014), most universities published all the documentation necessary for treatment and analysis. Nevertheless, in relation to some universities, and despite best efforts to obtain further documentation either through websites or direct contact with staff and documentation centers, it was confirmed that only a limited number of documents were actually published or made available.

In order to overcome this hindrance and to complete and/or deepen the analysis, if possible, an investigation will be pursued through interviews with the persons in charge of sustainability integration in each university to assess what has been done to implement ESD during DESD and what is being done at the present to propose strategies and policies for sustainability improvements and

to share them among all universities. It is expected that a more complete country profile for ESD implementation will emerge.

Based on the country profile developed in this research, each Portuguese university could share with all stakeholders (teaching staff, students, and community) all the initiatives and (best) practices in order to increase knowledge of the work that has been done, namely in terms of partnerships, fundraising, and other actions implementing sustainability. A platform— SharingSustainability4U—sharing sustainability initiatives based on this partnership idea is suggested.

This may be widened to a European or even to a worldwide platform, as universities are not all at the same stage concerning ESD. In the near future, this platform could be a worldwide reference for all universities to share and communicate activities, projects, and results concerning their ESD implementation. From there, the policies and strategies of multiple universities may be designed towards the implementation of ESD.

**Author Contributions:** C.F. had the idea for the paper. All the authors read, revised, and approved the final manuscript.

**Funding:** This research was funded by Portuguese Science and Technology Foundation (FCT) for its funding to the Centre for Environmental and Sustainability Research (CENSE), NOVA School of Science and Technology, NOVA University of Lisbon, grant number (UID/AMB/04085/2019).

**Acknowledgments:** We are grateful to Madalena Carvalho from documentation Centre in Universidade Aberta, which facilitated the contacts with other Public HEIs´ librarians that beforehand were not so fruitful in our numerous attempts.

Thanks also to the all the librarians for sharing their valuable but necessarily different experiences regarding research.

**Conflicts of Interest:** The authors declare no conflicts of interest.

**Appendix A**

**Figure A1.** Distribution map of Portuguese public universities. Scale: 1:65,000 km; source: CRUP, 2018. Remark: The acronyms are not the ones by which the HEIs are generally known.

**Figure A2.** Type of document by each public university regarding the Pareto graph.

#### **References**


© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

#### *Article*

## **Reform of Chinese Universities in the Context of Sustainable Development: Teacher Evaluation and Improvement Based on Hybrid Multiple Criteria Decision-Making Model**

## **Sung-Shun Weng 1, Yang Liu 1,\* and Yen-Ching Chuang <sup>2</sup>**


Received: 13 July 2019; Accepted: 29 September 2019; Published: 2 October 2019

**Abstract:** China is pushing universities to implement reforms in order to achieve the sustainable development goals, but with the development level of teachers becoming the key restricting factor. In this sense, teacher evaluation and improvement act as positive factors for China to achieve the 2030 sustainable development goals. Previous studies on teacher evaluation have usually assumed that the relationship between the evaluation criteria is independent, with the weights of each standard derived from this assumption. However, this assumption is often not in line with the actual situation. Decisions based on these studies are likely to waste resources and may negatively impact the efficiency and effectiveness of teachers' sustainable development. This study developed an integrated model for the evaluation and improvement of teachers based on the official teacher evaluation criteria of China's International Scholarly Exchange Curriculum (ISEC) programme and a multiple criteria decision-making methodology. First, a decision-making trial and a laboratory-based analytical network process were used to establish an influential network-relation diagram (INRD) and influential weights under ISEC standards. Next, an important performance analysis was used to integrate the weight and performance of each standard to produce a worst-performance criterion set for each university teacher. Finally, the worst performance set used an INRD to derive an improvement strategy with a cause–effect relationship for each teacher. This study chose a Chinese university that has implemented teaching reform for our case study. The results show that our developed model can assist decision-makers to improve their current evaluations of teachers and to provide a cause–effect improvement strategy for education reform committees and higher education institutions.

**Keywords:** sustainable development; International Scholarly Exchange Curriculum (ISEC) standards; university teacher evaluation and improvement; multiple criteria decision-making (MCDM); decision-making trial and evaluation laboratory (DEMATEL); DEMATEL-based analytical network process (DANP); importance-performance analysis (IPA)

#### **1. Introduction**

Education is crucial to sustainable development. The action plan entitled 'Transforming our World: The 2030 Agenda for Sustainable Development' jointly concluded among 193 countries was released at the World Summit on Sustainable Development in September 2015 [1]. Since then, the goal of sustainable development has become a new goal of global development. One of the goals in the agenda is about ensuring inclusive and equitable quality education and lifelong learning opportunities for all [2,3]. The Talloires Declaration announced in 1990 pioneered the inclusion of sustainable development in

higher education, highlighting the role of higher educational institutions in promoting global sustainable development [3]. The higher educational institutions subsequently began to shoulder the task of training human resources with the vision of sustainable development. Following the identification of the Global Education Roadmap 2030 at the World Education Forum 2015, the Global Action Program on Education for Sustainable Development was launched. Sustainable development and education for sustainable development thus gained stronger momentum to be promoted worldwide [4]. According to UNESCO, education for sustainable development can guarantee the future of the economy, the environment, and society. To achieve the goal, universities are undergoing teaching and research reforms [5]. China published the National Plan on Implementation of the 2030 Agenda for Sustainable Development in September 2016. The Plan conveyed China's resolution to advance the said 2030 Agenda [6]. A sub-plan was proposed to fulfil the educational development goal, further facilitating the reform of China's higher educational institutions from the perspective of sustainable development. In 2019, China's Ministry of Education (MOE) came up with the plan to develop nearly 10,000 national top and 10,000 provincial top majors for the undergraduates, thus boosting the reform of teaching activities in higher education institutions [7].

Teachers play a fundamental role in the reforms [3,8]. While higher education institutions are carrying out reforms in response to the goal of sustainable development, sustainability-based teaching activities pose new challenges to teachers at universities. The Teaching Staff Development Plan is a feasible approach to accelerating the integration of education for sustainable development [5,9]. For China's universities, particularly the regional ones (i.e., universities established by governments under the provincial level), the quality of teachers has become a key obstacle to reform. Therefore, in 2018, the Action Program to Rejuvenate Education for the Teaching Staff 2018–2022 was released by five ministries in China, including the MOE [10]. In this context, regional universities began to invest heavily in the training and re-education of the teaching staff, in the hope that they would stand out in the competition that takes sustainable development as its goal.

However, a key defect with China's higher education is the lack of, and uneven distribution of sufficient quality education resources [11,12]. At the same time, existing education resources have not been fully leveraged yet [13,14]. Among such resources, there is an important issue of university teacher development wherein massive amounts of invested resources are inefficiently used. Unless addressed properly, this issue would hinder universities in China from realizing the goal of education for sustainable development. If teacher evaluations can identify the core reasons why and where they perform poorly before resources are invested in specific areas, the information can be used to effectively improve the resource investment and use rate in Chinese universities. To solve this problem, a subsidiary department of the Chinese MOE developed the International Scholarly Exchange Curriculum (ISEC) program based on the directive of "globalizing education to deepen reforms in higher education". University teacher evaluation and improvement is part of the ISEC program, where the assessment criteria of university teachers are also known as the ISEC standard in this study.

Teacher evaluation as a management tool of education aims to facilitate the growth of the teaching staff [15–18] and are within the scope of multiple criteria decision-making (MCDM). For example, Ghosh [19] combined the analytic hierarchy process (AHP) and the technique for order of preference by similarity to ideal solution (TOPSIS) in order to evaluate faculty performance in engineering education. Pavani et al. [20] developed an expert-based group model for evaluating teacher performance using fuzzy AHP and TOPSIS. Xu et al. [21] evaluated teaching performance on a smart campus. Wang et al. [22] proposed a hybrid model for classroom teaching performance based on TOPSIS and the triangle fuzzy number. These studies provided valuable contributions to teacher evaluations. However, the relationships among the criteria in these models are independent and do not reflect real conditions. To address this issue, the suggestions for teacher improvement provided by these MCDM models are often directed toward the improvement of poor performance, but the factors that lead to poor performance may not always be addressed by these improvement strategies. As the

Chinese proverb goes, "treat the head when the head aches, treat the foot when the foot hurts". Thus, the improvement of the teaching capacity is not satisfactory, resulting in resource wastage and scarcity.

To address this issue, the study combined the MCDM with the teacher evaluation standards adopted by ISEC in order to create a new evaluation model. First, the evaluation model is the use of China's ISEC standards. These standards in China constitute the practical application of ISEC teachers to choose and improve problems. Based on the ISEC standards, the decision-making trial and evaluation laboratory (DEMATEL)-based analytic network process (ANP; together DANP) was then used to establish an influential network-relation diagram (INRD), and subsequently to obtain the influential weights within ISEC standards. The derived INRD has been proven to be an effective tool to explore the cause and effect relations in many papers [23–27]. Second, the importance-performance analysis (IPA) was used to combine the influential weights and performance of attributes, and to capture a set of criteria for identifying where each teacher performs worst. Finally, decision-makers can use the INRD to focus on the causation of poor performance to determine the actual factors related to each teacher's performance, and to prioritise a direction for improvement. The integrated model of DEMATEL and IPA methods has been successfully applied to different studies [28–32]. The model focuses on analyzing the factors influencing poor teacher performance according to certain criteria with the intention of using fewer but more focused resources to produce effective improvements. This method provides a new mechanism for the sustainable development of university teachers based on evaluations and also supplements the inadequacies of the existing studies to a certain degree.

In this study, empirical data from 15 domain experts from the ISEC management institute were applied to demonstrate our proposed model. The results show that "Professional ethics and literacy (*C*1)" is the primary influential standard and "Teacher ability and development (*C*2)" has the highest influential weight. Teachers *A* and *B* both performed poorly in "Teaching performance (*C*31)" and "Research cooperation (*C*32)". In other words, teachers *A* and *B* must improve their performance in these two standards. According to traditional performance improvement strategies, the ISEC management institute should invest resources in encouraging teachers to publish their teaching results and promote research cooperation among teachers. In practice, teacher *A* and teacher *B* enjoy the same training and development resources. However, the essential causes of their under-performance vary, which, accordingly, requires different training resources. This phenomenon reveals the issue of the untargeted allocation of teaching staff development resources in the reform of China's universities.

The rest of this paper is structured as follows. Section 2 provides a brief introduction of current teacher evaluations and outlines the literature documenting research methods. Section 3 introduces the DANP and IPA methods used in our new model. Section 4 details the implementation of ISEC topics in this model. Section 5 discusses the results and features of the model, and Section 6 summarizes our contributions and directions for future research.

#### **2. Review of University Teacher Evaluation Models**

The previous university teacher evaluation models can be roughly divided into three research stages: (1) the selection of appropriate criteria in the evaluation model, (2) building the decision-making model using the MCDM methodology, and (3) building the decision-making model using statistical or data analysis methodologies.

#### *2.1. Selection of Appropriate Criteria in the Evaluation Model*

This stage of research is focused on the selection of subjective and objective indicators in an evaluation model. Contradictions and conflicts exist between indicators, such as encouraging teacher vision and personal development versus increasing wages, and between objective evaluation results and critical feedback [33–35]. For example, when Mills and Hyles [36] evaluated university teachers at Oklahoma State University's Stillwater College (Stillwater, OK, USA), the results provided limited feedback and drawbacks included scattered goals, unclear standards, and inconsistencies in management perspectives. Based on this, they used interviews and surveys to understand opinions

on teacher evaluations, and their development process established a consolidated solution with a hierarchical architecture that integrated goals, directions, and procedures for annual performance evaluations. For this reason, some studies began to explore the establishment of more reasonable evaluation indicators and weights. For example, Desselle et al. [37] studied university teacher evaluation systems at Duquesne University's Mylan School of Pharmacy (Stillwater, OK, USA). They used a modified Delphi procedure to confirm 29 teaching activities and 44 academic actions, including their weights, in an evaluation standard. Filipe et al. [38] provided guidelines for avoiding conflict between processes and goals by developing a message management system to evaluate teaching activity—however, they encountered difficulties similar to those of previous studies due to differing interpretations of how to implement and assess teaching. Subjective indicators are an inevitable aspect of the assessment process, creating ambiguity that hinders the transparency and fairness of teaching performance assessments. However, some studies have shifted from subjective indicators that rely on the evaluator (e.g., enthusiasm for one's work) to relatively objective indicators (e.g., the number of reference papers). Although this can prevent fuzziness due to subjectivity, it results in fewer facets of evaluation that cannot fully reflect a teacher's true performance level [39–41]. Based on these lessons, others have proposed to combine decision analysis models with objective and subjective indicators or mathematical programming models, but they often lack real-time scoring systems based on theoretical rationality or have weight settings that vary depending on the evaluator's subjectivity. This neglects the principle of value trade-off and results in a total evaluation score that may have no substance or value. Using a methodology for systematic decision modelling to establish the factors and their weights in an evaluation system is a key problem that remains unsolved [42–44].

#### *2.2. Building Decision-Making Models Using MCDM Methodology*

The MCDM is specifically applied to solve evaluative decision-making problems with multiple criteria, i.e., evaluation, selection, and improvement problems. Some researchers have used MCDM methods to build decision models. For example, Ghosh [29] used AHP and TOPSIS to review teacher evaluations in engineering schools. Filipe et al. [38] developed a multi-criteria information system to review teaching practices. Hein et al. [45] used tools such as consensus theory, information entropy, and TOPSIS to construct a multi-criteria decision analysis method to evaluate 56 university professors. However, people's representations of their opinions with regard to objects or events, in reality, contain fuzziness. Therefore, some scholars have developed various fuzzy-based MCDM models. For example, Chen et al. [46] proposed a framework for teaching evaluation based on a combination of fuzzy AHP and fuzzy comprehensive evaluation methods. Chang and Wang [47] proposed a type of multi-criteria decision-making model oriented toward teachers in an attempt to solve the issues of pervasive subjectivity, imprecision, and fuzziness within the faculty. Dey Mondal and Ghosh [48] used AHP, fuzzy AHP, complex proportional assessment of alternatives with grey relations (COPRAS), and TOPSIS in combination with game theory and compromise planning methods in MCDM to evaluate the performance level of teachers. These models provide decision-makers with a simple and easy-to-use method for evaluating and selecting university teachers. However, in these models, decision-making is based on independent relationships among criteria that cannot provide decision-makers with suitable systemic improvement strategies for all university teachers.

#### *2.3. Building Decision-Making Models Using Statistical or Data Analysis Methodologies*

The last research stage has involved overcoming the fuzzy defects of these MCDM models that use statistical or data analysis methods to construct decision-making models. For example, Nikolaidis and Dimitriadis [49] established a framework based on statistical quality control to use student feedback to a maximum degree. Lyde et al. [50] used a multisource method for evaluation (MME) and improved constraints such as the timing of reflections, accountability from year to year, and mentoring in order to construct a more comprehensive formative teaching assessment tool. Bi [51] evaluated five years of teaching at a management school of a university by creating a mean and standard deviation diagram based on statistical process control theory. Xu et al. [31] used principal component analysis (PCA) to calculate and identify six primary components and then used AHP to calculate the weight of each hierarchy before using grey correlation to improve the TOPSIS target decision analysis algorithm to avoid errors in decision-making due to subjective factors. These models offer a perspective on data behaviour as a basis for decision-making, but they rely on massive amounts of data and are unable to provide decision-makers with causal influence relationships affecting teacher performance.

#### *2.4. Research Gaps in Their Decision-Making Models*

Past decision-making models presented different contributions to teacher evaluation, selection, and improvement problems. However, these MCDM models have a major defect in that the relationship between criteria in the evaluation model is independent. Therefore, current MCDM models cannot help decision-makers to obtain a guide for the performance improvement of each teacher. To fill the research gap, this study developed a novel MCDM model that uses the DANP method to construct the INRD and influential weights for criteria and the IPA method to search the worst performances of criteria for each teacher. The detailed modelling process and its corresponding method are described in Section 3. The comparison of the three categories of decision-making models is shown in Table 1.



#### **3. Our Proposed Hybrid DANP-IPA Model**

This study developed an integrated hybrid MCDM model that combines the DANP method and IPA analysis. The former can be used to derive the influential network-relationship diagram (INRD) and influential weights that can help decision-makers to understand the cause-effect direction based on a systemic perspective. The latter can help decision-makers to easily capture the worst performance of each teacher in all attributes. Finally, the worst performance attributes of each teacher can be based on the INRM to develop a series of the most appropriate improvement strategies. The modelling flow diagram and corresponding methods of this hybrid DANP-IPA model are depicted in Figure 1.

**Figure 1.** Modelling flow diagram of the decision-making trial and evaluation laboratory (DEMATEL)-based analytic network process (DANP)-importance–performance analysis (IPA) model.

#### *3.1. DANP Method*

The DANP method was developed by Lee et al. [52] by combining the DEMATEL technique [53] and the ANP method [54]. The DANP method retains interdependent relationships among criteria and further derives an influential network-relation diagram (INRD) and influential weights for all criteria. The INRD established by the DANP method can help to form decision-making equations for various systemic plans to improve alternative/objective performance after evaluation [55]. Based on this advantage, the method has been applied in many areas, such as public open space development [56], creative communities [57], quality of life [58], supplier management [59], airline performance [60], green buildings [61], and international airports [62]. The detailed steps in the DANP calculation are as follows.

#### Step 1: Build an initial influence–relationship matrix.

For an evaluation criteria model, respondents assess the degree of influence between criteria using a pairwise comparison based on a five-point Likert scale (ranging from 0 = "no influence" to 4 = "extremely high influence"). Then, the influential matrices of all respondents are integrated into a matrix by averaging to produce the initial influence relationship matrix *D*. Matrix *D* represents the actual experience within the group of all respondents:

$$D = \left[ \left( \sum\_{i \ominus = 1}^{n} c\_{ij}^{\ominus} \right) / a \right]\_{n \times n} \tag{1}$$

where *c*<sup>Θ</sup> *ij* is the result of the respondent Θ, indicating the degree of influence between criteria *i* and *j*; α is the total number of respondents, and *n* is the total number of criteria.

Step 2: Derive a normalized influence–relationship matrix.

The initial–influence relationship matrix *D* derives a normalized influence-relationship matrix *A* using Equations (2) and (3), in which all diagonal terms are 0 and the maximum sum of a row or column is 1:

$$\rho = \max\_{i,j} \left[ \max\_{} \sum\_{j=1}^{n} d\_{ij}, \max\_{} \sum\_{i=1}^{n} d\_{ij} \right] \tag{2}$$

$$\mathbf{A} = \mathbf{C} / \rho \tag{3}$$

where ρ is the maximum value of the sum of a row or column.

Step 3: Obtain a total influence–relationship matrix.

Matrix *A* calculates and adds the influence degree of each iteration through the Markov chain process and produces a total influence relation matrix *Q*, as shown in Equation (4):

$$\mathbf{Q} = \mathbf{A} + \mathbf{A}^2 + \dots + \mathbf{A}^\delta = \mathbf{A}(I - \mathbf{A})^{-1}, \text{ when } \lim\_{\delta \to \infty} \mathbf{A}^\delta = [0]\_{n \times n} \tag{4}$$

Step 4. Build an influential network–relationship diagram (INRD).

First, the sum of each row and column can obtain vectors *ui* and *vi* through Equations (5) and (6). Then, (*ui* + *vi*) is the total strength of influences given and received, or prominence, as shown in Equation (7). Otherwise, (*ui* − *vi*) is the net influence degree between given and received influences, also called the cause/effect. A positive cause/effect value indicates that factor *i* affects other factors and belongs to the cause group; if the value is negative, factor *i* is affected by other factors and belongs to the effect group. Finally, the INRD is established based on the vectors of prominence and cause/effect:

$$u\_i = (u\_i)\_{n \times 1} = \left[ \sum\_{j=1}^n q\_{ij} \right]\_{n \times 1}, \ i \in \{1, 2, \dots, n\} \tag{5}$$

$$v\_i = (v\_j)'\_{1 \times n} = \left[\sum\_{i=1}^n q\_{ij}\right]\_{1 \times n}, \quad j \in \{1, 2, \dots, n\} \tag{6}$$

where denotes transposition, *ui* indicates the sum of direct and indirect effects of the factor *i* on the other factors, and *vi* indicates the sum of direct and indirect effects factor *i* received from the other factors.

Step 5: Transfer to an unweighted supermatrix.

First, the total influence–relation matrix *<sup>Q</sup>* can be divided into two matrices: the attribute level *<sup>Q</sup><sup>C</sup>* and the dimension level *<sup>Q</sup>D*. Second, each row within a dimension in the total influence relation matrix *<sup>Q</sup><sup>C</sup>* uses Equations (7)–(9) to obtain the normalized total influence relation matrix *<sup>Q</sup>*<sup>ρ</sup> *<sup>C</sup>*, as shown in Equation (7), in which *Q*ρ<sup>11</sup> *<sup>C</sup>* is an example to demonstrate the basic concept of normalizing, as shown in Equations (8) and (9).

*Q*<sup>ρ</sup> *<sup>C</sup>* = *D*1 . . . *Di* . . . *D<sup>m</sup> c*11 *<sup>c</sup>* <sup>12</sup> . . . *<sup>c</sup>* <sup>1</sup>*m*<sup>1</sup> . . . *ci*1 *c i*2 . . . *<sup>c</sup> imi* . . . *cm*<sup>1</sup> *cm*<sup>2</sup> . . . *cmmm D*<sup>1</sup> *Dj Dm <sup>c</sup>*11···*c*1*m*<sup>1</sup> ··· *cj*<sup>1</sup>···*cjmj* ··· *cm*<sup>1</sup>···*cmmm* ⎡ ⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎣ *Q*ρ<sup>11</sup> *<sup>c</sup>* ··· *<sup>Q</sup>*ρ1*<sup>j</sup> <sup>c</sup>* ··· *<sup>Q</sup>*ρ1*<sup>m</sup> c* . . . . . . . . . *Q*ρ*i*<sup>1</sup> *<sup>c</sup>* ··· *<sup>Q</sup>*ρ*ij <sup>c</sup>* ··· *<sup>Q</sup>*ρ*im c* . . . . . . . . . *Q*ρ*m*<sup>1</sup> *<sup>c</sup>* ··· *<sup>Q</sup>*ρ*mj <sup>c</sup>* ··· *<sup>Q</sup>*ρ*mm c* ⎤ ⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦ *<sup>n</sup>*×*n*|*m*<*n*, *<sup>m</sup> <sup>j</sup>*=<sup>1</sup> *mj*=*<sup>n</sup>* (7)

$$q\_{i}^{11} = \sum\_{j=1}^{m\_{1}} q\_{ij}^{11}, i = 1, 2, \dots, m\_{1} \tag{8}$$

$$\mathbf{Q}\_{\mathbf{C}}^{\text{q11}} = \begin{bmatrix} q\_{11}^{11}/q\_{1}^{11} & \cdots & q\_{11}^{11}/q\_{1}^{11} & \cdots & q\_{1m\_{1}}^{11}/q\_{1}^{11} \\ \vdots & \vdots & & \vdots \\ q\_{i1}^{11}/q\_{i}^{11} & \cdots & q\_{ij}^{11}/q\_{i}^{11} & \cdots & q\_{im\_{1}}^{11}/q\_{i}^{11} \\ \vdots & & \vdots & & \vdots \\ \vdots & & & \vdots \\ q\_{m\_{1}1}^{11}/q\_{m\_{1}}^{11} & \cdots & q\_{m\_{1}1}^{11}/q\_{m\_{1}}^{11} & \cdots & q\_{m\_{1}1}^{11}/q\_{m\_{1}}^{11} \end{bmatrix} = \begin{bmatrix} q\_{11}^{011} & \cdots & q\_{1j}^{a11} & \cdots & q\_{1m\_{1}}^{a11} \\ \vdots & \vdots & & \vdots \\ q\_{i1}^{a11} & \cdots & q\_{ij}^{a11} & \cdots & q\_{im\_{1}}^{a11} \\ \vdots & & \vdots & & \vdots \\ q\_{m\_{1}1}^{a11} & \cdots & q\_{m\_{1}1}^{a11} & \cdots & q\_{m\_{1}1}^{a11} \end{bmatrix} \tag{9}$$

Lastly, the normalized influence relation matrix *Q*<sup>ρ</sup> *<sup>c</sup>* is transposed to obtain the unweighted supermatrix *B* = (*Q*<sup>ρ</sup> *c* ) , as shown in Equation (10):

*B* = (*Q*<sup>ρ</sup> *C*) = *D*1 . . . *Dj* . . . *D<sup>m</sup> c*11 *<sup>c</sup>* <sup>12</sup> . . . *<sup>c</sup>* <sup>1</sup>*m*<sup>1</sup> . . . *cj*1 *c j*2 . . . *<sup>c</sup> jmj* . . . *ci*1 *ci*2 . . . *cmmm D*<sup>1</sup> *Di Dm <sup>c</sup>*11···*c*1*m*<sup>1</sup> ··· *ci*<sup>1</sup>···*cimi* ··· *cm*<sup>1</sup>···*cmmm* ⎡ ⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎣ *<sup>B</sup>*<sup>11</sup> ··· *<sup>B</sup>i*<sup>1</sup> ··· *<sup>B</sup>m*<sup>1</sup> . . . . . . . . . *<sup>B</sup>*1*<sup>j</sup>* ··· *<sup>B</sup>ij* ··· *<sup>B</sup>mj* . . . . . . . . . *<sup>B</sup>*1*<sup>m</sup>* ··· *<sup>B</sup>in* ··· *<sup>B</sup>mm* ⎤ ⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦ *<sup>n</sup>*×*n*|*m*<*n*, *<sup>m</sup> <sup>j</sup>*=<sup>1</sup> *mj*=*<sup>n</sup>* (10)

Step 6: Obtain a weighted supermatrix.

Each row within the goal in the total influence–relation matrix *<sup>Q</sup><sup>D</sup>* uses Equations (11)–(13) to obtain the normalized total influence relation matrix *Q*<sup>ρ</sup> *<sup>D</sup>*, as shown in Equation (11). Matrices *<sup>Q</sup>*<sup>ρ</sup> *<sup>C</sup>* and *Q*<sup>ρ</sup> *<sup>D</sup>* produce a new matrix through Equation (14), the weighted supermatrix *<sup>B</sup>*Θ:

$$\mathbf{Q}\_D = \begin{bmatrix} q\_{11} & \cdots & q\_{1j} & \cdots & q\_{1m} \\ \vdots & & \vdots & & \vdots \\ q\_{i1} & \cdots & q\_{ij} & \cdots & q\_{im} \\ \vdots & & \vdots & & \vdots \\ q\_{m1} & \cdots & q\_{mj} & \cdots & q\_{mm} \end{bmatrix}\_{\text{nt}\times\text{nt}} \tag{11}$$

$$d\_i = \sum\_{j=1}^{m} q\_{D'}^{ij} \\ i = 1, 2, \dots, m \text{ and } q\_{D}^{\text{qij}} = q\_{D}^{\text{ij}} / d\_{i\cdot} \\ j = 1, 2, \dots, m \tag{12}$$

$$\mathbf{W} = (\mathbf{Q}\_{\mathcal{D}}^{\boldsymbol{\rho}})' = \begin{bmatrix} f\_{\mathcal{D}}^{11}/d\_{1} & \cdots & f\_{\mathcal{D}}^{1j}/d\_{1} & \cdots & f\_{\mathcal{D}}^{1m}/d\_{1} \\ \vdots & & \vdots & & \vdots \\ f\_{\mathcal{D}}^{11}/d\_{i} & \cdots & f\_{\mathcal{D}}^{1j}/d\_{i} & \cdots & f\_{\mathcal{D}}^{1m}/d\_{i} \\ \vdots & & \vdots & & \vdots \\ f\_{\mathcal{D}}^{m1}/d\_{m} & \cdots & f\_{\mathcal{D}}^{mj}/d\_{m} & \cdots & f\_{\mathcal{D}}^{mm}/d\_{m} \end{bmatrix}\_{\text{sym}} = \begin{bmatrix} f\_{\mathcal{D}}^{11} & \cdots & f\_{\mathcal{D}}^{1j} & \cdots & f\_{\mathcal{D}}^{1m} \\ \vdots & & \vdots & & \vdots \\ f\_{\mathcal{D}}^{q1} & \cdots & f\_{\mathcal{D}}^{qlj} & \cdots & f\_{\mathcal{D}}^{qlm} \\ \vdots & & \vdots & & \vdots \\ f\_{\mathcal{D}}^{qm1} & \cdots & f\_{\mathcal{D}}^{mqj} & \cdots & f\_{\mathcal{D}}^{mm} \end{bmatrix}\_{\text{norm}} \tag{13}$$

$$\mathbf{B}^{\Theta} = \mathbf{W} \times \mathbf{B} = \begin{bmatrix} \vdots & \vdots & \vdots \\ q\_D^{m1}/d\_m & \cdots & q\_D^{mj}/d\_m & \cdots & q\_D^{mm}/d\_m \end{bmatrix}\_{\text{new}} \begin{bmatrix} \vdots & \vdots & \vdots & \vdots \\ q\_D^{m1} & \cdots & q\_D^{mj} & \cdots & q\_D^{mm} \\ \vdots & \cdots & q\_D^{m1} \times \mathbf{B}^{11} & \cdots & q\_D^{mm} \times \mathbf{B}^{m1} \\ \vdots & \vdots & & \vdots \\ q\_D^{p1j} \times \mathbf{B}^{1j} & \cdots & q\_D^{njj} \times \mathbf{B}^{ij} & \cdots & q\_D^{mmj} \times \mathbf{B}^{mj} \\ \vdots & \vdots & & \vdots \\ q\_D^{p1m} \times \mathbf{B}^{1m} & \cdots & q\_D^{mm} \times \mathbf{B}^{int} & \cdots & q\_D^{mm} \times \mathbf{B}^{mm} \end{bmatrix} \tag{14}$$

Step 7: Limit the weighted supermatrix and derive the influential weights.

The weighted supermatrix *B*<sup>Θ</sup> convergences the influence degree of each time through the Markov chain process and finally obtains the influential weights for all criteria/dimensions, as shown in Equation (15).

$$\lim\_{\Lambda \to \infty} \left( \mathcal{B}^{\Theta} \right)^{\Lambda} \tag{15}$$

#### *3.2. Importance–Performance Analysis Method*

Importance–performance analysis (IPA) is a well-known business management method [63] that was first developed by Martilla and James [64] to identify the critical performance criteria of products or services [65,66]. The method is used to create an IPA matrix or priority map using standard performance and importance scores, which can be divided into four quadrants (Q1–Q4), as shown in Figure 2 [31,63,67]:


This approach can help decision-makers to easily understand the performance and importance of criteria. The method is widely used in many different areas, such as supplier management [31], tourism development [66,68], and strategy management [69].

**Figure 2.** Importance-performance analysis (IPA) map.

#### **4. Empirical Case**

In this section, an empirical study using data from the ISEC in China is presented to illustrate the application of the proposed DANP-IPA model for evaluating and improving the performance of university teachers.

#### *4.1. Case Background Problem Description*

Compared with developed countries, China lacks higher education resources and their distribution is imbalanced, especially at regional universities, which constitute the majority of universities in China. To solve this problem, a subsidiary department of the Chinese MOE developed the International Scholarly Exchange Curriculum (ISEC) program based on the directive of "globalizing education to deepen reforms in higher education". Universities that participated in the program received assistance with curriculum, teaching, and quality assurance reforms—however, these reforms were based on teachers. For this reason, ISEC sought to establish a sustainable development mechanism in order to cultivate quality teaching teams that would autonomously reform curriculums and teaching and achieve the goal of comprehensive teaching reform. This line of thinking is a departure from China's current method of promoting change from the top downward. The core of this development program involves teachers and the sustainable development of promoting reforms in higher education. The corresponding content includes (1) curriculum systems, (2) support for teacher development systems, and (3) service systems. Currently, the program has been implemented in approximately 30 test universities in nine provinces with more than 1500 teachers listed in ISEC, accumulating practical experience with both successes and failures. These ISEC teachers have met the ISEC inclusion criteria, i.e., age, academic qualifications, foreign study or exchange experience, international curriculum teaching experience, critical thinking, and ISEC mission acceptance. The program prioritizes the development of teachers on the front line as its core. ISEC is the coordinating mechanism and training platform for teacher development, partnering with universities to use global educational resources to support the improvement of teacher abilities. Next, ISEC teachers act as leaders to push teaching reforms throughout the university and faculty. Therefore, establishing a decision-making model with

practical value to assist decision-makers with effectively evaluating ISEC teachers and improving their abilities is a critical problem.

With assistance from the ISEC management institute, 15 ISEC domain experts were selected from the ISEC expert database (one American, two Australian, and 12 Chinese). These experts were teachers, administrative staff members, or ISEC teacher representatives (associate professor or above) at renowned universities. The ISEC domain experts represent elite ISEC teachers. As such, this study is based on the ISEC university teacher standard indicator system (Table 2), and it integrated MCDM and IPA tools to establish a mixed multi-criteria decision-making model. This DANP-IPA model can be used to evaluate and improve ISEC standards for teachers.



#### *4.2. INRD and Influential Weight Using the DANP Method*

The degree of influence between standards was calculated using a five-point measurement scale for all experts, and then Equation (1) was used to consolidate and obtain an initial influence–relationship matrix (Table 3). In this matrix, Equations (2)–(4) allowed the inference of a total influence–relationship matrix (Table 4). Using Equations (5) and (6), the influence structure of each standard was obtained (Table 5) to draw an INRD (Figure 3).




**Table 4.** Total influence–relation matrix.

**Table 5.** Sum of given influence (*ri*) and received influence (*di*).


**Figure 3.** Influential network-relation diagram (INRD) of ISEC teacher standard evaluation system.

Figure 3 shows the entire mutual influence network within the ISEC teacher standard evaluation system, where "Professional ethics and literacy (*C*1)" is the primary standard influencing "Teacher ability and development (*C*2)" and "Teacher performance and contributions (*C*3)"—this shows that "Professional ethics and literacy (*C*1)" is the basis of two criteria. That is, a teacher's professional ethics and literacy impact their abilities and future development, which is reflected in performance and teaching contributions. Further analysis showed that "Professional ethics (*C*11)" and "Teaching implementation (*C*22)" are causal groups (i.e., *ri* − *di* > 0). "Research cooperation (*C*32)" and "Teaching performance (*C*31)" are effect groups (i.e., *ri* − *di* < 0). Past studies focused on decision-makers investing resources to improve specific standards to correct poor performance. However, this may not address the true cause of the problems as decision-makers may neglect the influence structure of standards being composed of interdependent, not independent, relationships. When decision-makers focus solely on poorly performing standards (effect) and invest massive amounts of resources, they do not recognize that the problem may stem from causal standards. For instance, "Research cooperation (*C*32)" maybe a teacher's poorest performing standard and the decision-maker may hope that the teacher can cooperate with other researchers. They then host workshops to provide cooperative opportunities and invest massive amounts of resources to encourage teachers to work together on research projects. However, teachers may want to focus on lectures and teaching due to their own state of "Professional ethics (*C*11)" and "Professional literacy (*C*12)", which is conservative. They may simply lack an assertive attitude toward learning, leading to poor research ability. In other words, various factors can cause "Research cooperation (*C*32)" to be the area of poorest performance. Here, INRD provides a systemic view that assists decision-makers in understanding the relationship structure influencing the standards to pinpoint each teacher's problems. The aspect of "Basic teaching skills (*C*21)" in "Teacher ability and development" is a basic standard in teaching because it influences other standards such as "Teaching implementation (*C*22)", "Professional development (*C*24)", and "External review (*C*23)".

For the weights of the influencing relationships, the total influence–relationship matrix (Table 4) uses Equations (7)–(10) to establish an unweighted supermatrix (Table 6). Next, Equations (11)–(14) are used in the matrix to establish a weighted supermatrix (Table 7). Finally, Equation (15) is used to achieve a stable, extreme supermatrix. The influence weights for each standard are shown in Table 8.


**Table 6.** Unweighted supermatrix.

**Table 7.** Weighted supermatrix.



**Table 8.** Influence weights for each criterion of ISEC standard.

Table 8 shows the influential weight of each standard within the entire system and the degree of influence for each standard, which is beneficial for subsequent teacher evaluation processes, as the performance in each standard considers the degree of influence. Judging from the results, the dimension of "Teacher ability and development (*C*2)" has the highest influential weight and reflects teacher ability and development as the most influential relationship in the evaluation system. In addition to being driven by "Professional ethics and literacy (*C*1)", performance is also reflected by "Teacher performance and contributions (*C*3)". A teacher's long-term planning and development in each period of "Teacher performance and contributions (*C*3)" impacts improvements in future "Teacher ability and development (*C*2)", which is why "Teacher ability and development (*C*2)" is the most influential dimension. "Teaching performance (*C*31)" and "Research cooperation (*C*32)" are the top two criteria in terms of influential weight because they reflect performance in the other dimensions, so if other dimensions perform poorly, this is reflected in "Teaching performance (*C*31)" and "Teacher ability and development (*C*2)".

#### *4.3. University Teacher Evaluation Using the IPA Method*

Based on teacher performance and weights, this section outlines our use of the IPA method to analyze and gather the standard in which each teacher performs the poorest. With the assistance of the ISEC management institute, five members of the review committee and three university teachers participated in this study. All five members had experience in evaluating ISEC teachers for more than one semester; they used a 0–10-point scale to evaluate three university teachers. These scores were averaged and consolidated into a performance score (Tables 9–11). Next, the centre values of weights and performance were used as threshold values to separate standards into four groups as follows: Group I—high weights and performance; group II—low weights and high performance; group III—low weights and performance, and group IV—high weights and low performance. Decision-makers must focus on group IV, as standards within the group are categorized as high weight but the performance in these standards is the poorest. The investment of resources should prioritize the standards in this group to effectively improve the performance in the group. The analysis results for the three university teachers are provided in Table 12 and Figures 4–6.

Figures 4–6 show that teacher *C* performed the best overall. Teachers *A* and *B* perform poorest in the standards of "Teaching performance (*C*31)" and "Research cooperation (*C*32)". In other words, teachers *A* and *B* must improve their performance in these two standards. Based on traditional performance improvement strategies, the ISEC management institute should invest resources to encourage these teachers to publish their teaching results in order to promote academic cooperation among teachers. However, their problems may not actually stem from these two standards because their performance simply reflects the existence of a problem. To avoid this issue, the cause–effect relationship analysis of INRD (Figure 3) can be used to understand the entire issue and propose appropriate improvement measures for each university professor—this is also the solution that allows

both the minimization of resources and the maximization of benefit. This will be discussed in further detail in the next section.


**Table 9.** The average performance of university teacher *A*.

**Table 10.** The average performance of university teacher *B*.


**Table 11.** The average performance of university teacher *C*.


**Table 12.** IPA method for university teacher evaluation.


Note: Center value as a threshold value (7.5, 0.125). The values of x and y are derived from the central point between the maximum and minimum of performance and weight, respectively.

**Figure 4.** IPA of teacher *A* using the standard evaluation system.

**Figure 5.** IPA of teacher *B* using the standard evaluation system.

**Figure 6.** IPA of teacher *C* using the standard evaluation system.

#### **5. Discussion**

This section presents the results of the IPA with a cause-and-effect relationship analysis based on the INRD to create plans and strategies for teacher improvement. Finally, the influential weights produced from this study and actual current weights are explored.

#### *5.1. Proposal for Improvement Suggestions Based on the INRD*

To understand how INRD is used, this study used the case of teacher *B* to explain how INRD is used in analyzing and proposing improvement measures. Figure 7 shows that teacher *B* performed poorly in "Teaching performance (*C*31)" and "Research cooperation (*C*32)". Of the two, "Research cooperation (*C*32)" was associated with the poorest performance. However, the standards that impact "Teacher performance and contributions (*C*3)" are "Teacher ability and development (*C*2)" and "Professional ethics and literacy (*C*1)". The teaching-related standards are "Basic teaching skills (*C*21)", "Teaching implementation (*C*22)", and "External review (*C*23)", which all had good scores (7.6 to 8.2). This was also reflected in the "Teaching performance" score (7.2). Of these, "Professional development (*C*24)" was the poorest performing standard (6.0), which may be due to different factors, such as not listing research cooperation in professional development planning, using the majority of their daily time for teaching or administrative tasks, not having the ability or opportunity to conduct research cooperation, or not having a suitable research budget. All these factors can lead to poor performance in "Research cooperation (*C*32)". Based on the above analysis, this study conducted discussions with ISEC management staff and proposed suggestions for improvements based on the perspectives of "Professional ethics and literacy (*C*1)" and "Teacher ability and development (*C*2)". The university should reassess teacher *B*'s professional ethics and literacy, research ability, and time spent teaching, and adjust these three aspects, for instance, by improving a teacher's professional acknowledgment, adjusting their lecture and teaching time, arranging for them to learn the skills required for research, and arranging opportunities for research cooperation. The ISEC Management Institute could arrange a series of comprehensive courses to improve research capability and post-curricular meetings for research cooperation.

**Figure 7.** INRD for teacher *B*.

#### *5.2. Comparison of Weights*

Next, this section compared the weights obtained by the analysis of ISEC teacher standards using the DANP and AHP methods to actual weights currently being used, as shown in Table 13. This study discovered the following: (1) actual weights and DANP weights stem from the cumulative practical teaching experience of experts, and the results show that the rankings in both are close. This result shows that the establishment of actual weights contains implicit systemic perspectives. (2) Actual weights do not provide specific values but approximate values. For instance, the standards "Basic teaching skills (*C*21)" and "Professional development (*C*24)" are both weighted 0.2, whereas "Teaching implementation (*C*22)" and "External review (*C*23)" are both weighted 0.3. The same weight values provide the impression that the standards "Basic teaching skills (*C*21)" and "Professional development (*C*24)" are equally important, and "Teaching implementation (*C*22)" and "External review (*C*23)" are equally important. However, the DANP weights show that: "External review (*C*23)" "Professional development (*C*24)" "Teaching implementation (*C*22)" "Basic teaching skills (*C*21)". By comparison, precise weights allow for precise evaluations along with subsequent use and investment of resources for teacher development. From a practical standpoint, this does not merely involve categorizing the importance of standards. (3) Currently, actual weights do not precisely describe factors. Further, past studies have often used the AHP method as weight analysis of criteria in teacher evaluation models [19,20,29,46,48]. In this study, the ranking of dimensions and criteria between the AHP method and the other two methods is a little different, however, AHP weights can provide specific values for each criterion. However, the AHP method assumes that the relationship between the criteria is independent, which is inconsistent with the operation of the real world. On the contrary, the DANP method used to obtain the weights of the standards from the perspective of systemic influence allows these systemic perspectives to identify cause-and-effect relationships using INRD. This assigns greater significance and high explanatory power to standard weights. Based on this feature, the DANP has been successfully applied to different issues [56–62]. Compared to the previous model, the DANP method employed in this study can provide more specific information to help decision-makers to obtain a complete systematic solution.


**Table 13.** Comparative analysis of weights.

#### **6. Conclusions and Remarks**

ISEC teacher evaluation standards are a key aspect of Chinese education reform. In the past, the ISEC management institute has focused on teachers' poorest performing standards to propose a series of improvement measures. However, these measures did not consider the mutual influence structures amongst the standards. Therefore, the improvements often treat the symptoms but not the source, and resources are not used efficiently for maximum benefit. Standard weights are expressed in integers that cannot effectively differentiate the degree of relative importance among standards and

explain the management significance behind each weight. This type of evaluation process does not truly reflect the abilities of each teacher. To solve this issue, this study developed a mixed multi-criteria decision-making (DANP-IPA) model based on ISEC teacher standards.

First, this model provides the INRD and influential weights based on the systemic perspective. INRD assists decision-makers in understanding the influential relationship structure among standards. Influential weights integrate influential perspectives into subsequent processes of teacher evaluation so that improvement measures are based on cause and effect. In the practical case studies used in this research, INRD identified "Professional ethics and literacy (*C*1)" as the primary influential standard that impacts "Teacher ability and development (*C*2)" and "Teacher performance and contributions (*C*3)". In other words, the basic factors that affect teacher ability and performance are the individual's own professional ethics and literacy. "Teacher ability and development (*C*2)" was shown to have the highest influential weight because it is driven by "Professional ethics and literacy" and is reflected in "Teacher performance and contributions (*C*3)". A teacher's long-term planning and development and "Teacher performance and contributions (*C*3)" in each period were found to be influenced by future improvements in "Teacher ability and development (*C*2)". This is why "Teacher ability and development (*C*2)" was found to have the highest influential ranking of all dimensions. "Teaching performance (*C*31)" and "Research cooperation (*C*32)" were the two highest in terms of influential weight because they reflect the performance of other standards, meaning that if other standards perform poorly, this is eventually reflected in "Teaching performance (*C*31)" and "Research cooperation (*C*32)". Finally, the IPA performance analysis showed that teacher *C* had the best overall performance—in comparison, teachers *A* and *B* both performed poorly in "Teaching performance (*C*31)" and "Research cooperation (*C*32)". Teachers *A* and *B* must improve their performance in these two standards. According to traditional performance improvement strategies, the ISEC management institute should invest resources to encourage teachers to publish their teaching results and promote research cooperation among teachers. However, these problems may not actually stem from these two standards because they are simply a reflection of the problem.

Based on the above analysis, this study conducted discussions with ISEC management staff and provided suggestions for improvements based on the perspectives of "Professional ethics and literacy (*C*1)" and "Teacher ability and development (*C*2)". Specifically, the university should reassess teacher *B*'s professional ethics and literacy, research ability, and time spent teaching and adjust these three aspects, for example, by improving the teacher's professional acknowledgment, adjusting their lecture and teaching time, arranging for them to learn the skills required for research, and arranging opportunities for research cooperation. The ISEC management institute could arrange a series of comprehensive courses to improve research capability and post-curricular meetings for research cooperation. Therefore, this study contributes to this topic as follows: (1) our method can be combined with INRD (Figure 3) to analyze the cause-effect relationship and understand the entire problem to propose the most suitable improvement measures for each university professor. (2) Our solution both minimizes resources and maximizes benefits to improve the efficiency of resource investment in the development of university teachers. Our method will have a catalyzing effect on the continued development and cultivation of teachers.

Although this study provides a scientific decision-making model, the orientation of future research involves the exploration of multiple facets. This study focused on the establishment of the model and the application of ISEC teacher assessment and improvement. In our current research, the limitations are the ISEC evaluation standards and improvement strategies that must depend on domain-experts' knowledge (i.e., influential network-relation diagram and influential weights). The data source requires the expertise of a group of ISEC domain experts with practical experience in the issue area. Therefore, the data sources (i.e., DANP and performance) require support from the ISEC management institute. In addition, the student's perspective in the evaluation process and the subsequent development of the teacher were not included in the scope of this research, which forms another limitation of this study. Based on these limitations, one direction of future studies could involve the consideration of the

perspective of students in the evaluation process for university teachers. Another research opportunity could be the integration of research on data exploration and multi-criteria decision-making into the model with the use of big data to analyse and understand the correlations between standards for filtering key standards and their weights—as a way of ultimately proposing improvement measures based on expert knowledge with objective behavioural rules and subjective practical experience. As the Chinese education reform deepens, various problems relating to building decision-making models will require corresponding solutions from future researchers.

**Author Contributions:** Conceptualization, S.-S.W.; methodology, Y.-C.C.; numeration: Y.-C.C.; investigation, Y.L.; resources, Y.L.; data curation, Y.L.; writing—original article, Y.L.; writing—review and editing, S.-S.W. and Y.-C.C.; supervision, S.-S.W.; project administration, S.-S.W.

**Funding:** This research received no external funding

**Acknowledgments:** The authors would like to thank the management department of China's ISEC program for its strong support for this study. We are grateful to the editor of the Special Issue and the two anonymous reviewers for their constructive and helpful comments and suggestions.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **Spanish Universities' Sustainability Performance and Sustainability-Related R&D**+**I**

**Daniela De Filippo 1,2,\* , Leyla Angélica Sandoval-Hamón 1,3 , Fernando Casani 1,3 and Elías Sanz-Casado 1,4**


Received: 29 July 2019; Accepted: 8 October 2019; Published: 10 October 2019

**Abstract:** For its scope and the breadth of its available resources, the university system is one of the keys to implementing and propagating policies, with sustainability policies being among them. Building on sustainability performance in universities, this study aimed to: Identify the procedures deployed by universities to measure sustainability; detect the strengths and weaknesses of the Spanish university system (SUS) sustainability practice; analyse the SUS contributions to sustainability-related Research, Development and Innovation (R&D+I); and assess the efficacy of such practices and procedures as reported in the literature. The indicators of scientific activity were defined by applying scientometric techniques to analyse the journal (Web of Science) and European project (CORDIS) databases, along with reports issued by national institutions. The findings showed that measuring sustainability in the SUS is a very recent endeavour and that one of the strengths is the university community's engagement with the ideal. Nonetheless, high performance is still elusive in most of the items analysed. Whereas universities account for nearly 90 % of the Spanish papers published in the WoS subject category, Green and Sustainable Science and Technology, their contribution to research projects is meagre. A divide still exists in the SUS between policies and results, although the gap has been narrowing in recent years.

**Keywords:** university policies on sustainability; scientometric indicators; Web of Science; CORDIS; Spanish university system

#### **1. Introduction**

Universities as institutions make a significant social, economic, academic, scientific and technological contribution to their local and national environments. These contributions have been widely discussed in the literature [1–5] because universities are considered "the most prominent producers of fundamental knowledge, which has been argued to be one of the main drivers of economic growth" [6]. According to Eurostat [7], 19.6 million students and 1.5 million professors engaged in tertiary education in 2016, whilst public spending on higher education came to 1.2% of European GDP in 2015.

As those figures show, in light of its scope and the breadth of its available resources, the university system is a key agent in implementing and propagating all manner of policies, with sustainability being among them. Higher education institutions (HEIs) are consequently crucial to societal transformation [8].

One of the primary goals of the European Commission document, "Europe's 2020 Strategy for Smart, Sustainable and Inclusive Growth" is for 40% of the Union's population between the ages of 30 and 34 to have a post-secondary degree by the target year [9]. Higher education institutions, as knowledge and innovation centres, therefore play a significant role in furthering the migration to sustainable development models. At the same time, they are introducing change in their own processes to adapt to the new scenario, which impacts not only core education but research, institution management and community outreach [10].

As there are many higher education institutions, the world over has become increasingly aware of their impact on the environment. They have made substantial efforts to enhance their understanding of the environmental dimensions of their operations and the implications and impact of higher education activities [11]. HEIs therefore play a catalytic role in societies' engagement with sustainability. In light of that complex challenge, HEIs must pay greater heed to their internal and external modi operandi. This means that HEIs must realise that "the development must meet the needs of the present without compromising the ability of future generations to meet their own needs" [12]. In this line, the concept of a sustainable university is broad and includes consideration of "a sustainable university as a higher education institution, as a whole or as a part, that addresses, involves and promotes, on regional or global level, the minimization of environmental, economics, societal, and health negative effects in the use of their resources in order to fulfill its main functions of teaching, research, outreach and partnership, and stewardship among other as a way to helping society make the transition to sustainable life styles" [13]. Therefore, HEIs need to assume their responsibilities in education (ad hoc courses and curricula), research, on-campus operations and community outreach with greater integrity and transparency. This can be reflected, for instance, in resource allocation planning and commitment to sustainable development.

The push to make sustainability and sustainable development an overarching concern has become particularly prevalent since the last decades of the past century. Leal Filho, Brandli, Becke, Skanavis, Kounani, Sardi, and Raath have described the efforts of 35 universities (on five continents) to implement sustainability policies and procedures [14]. Other authors define a "sustainability or environmental policy" as "one element of HEIs' sustainability governance documents that also includes plans, strategies, and reports" [15].

Several national and international bodies have echoed these concerns and instituted sustainability-related statements, programmes, tools and systems. The Stockholm Declaration on the Human Environment (1972), the Tbilisi Declaration (1977) and the UNESCO's Magna Charta of European Universities (1988) constitute clear examples. Some of the most prominent programmes include the UN-supported Sustainable Development Solutions Network, the International Sustainable Campus Network (ISCN), the Association for the Advancement of Sustainability in Higher Education (AASHE) in the United States and the Environmental Association for Universities and Colleges (EAUC) in the United Kingdom. The rise in the number of scientific publications on the subject attests to growing research interest, prompting the Web of Science to establish a new subject category, Green and Sustainable Science and Technology (GSST), that groups journals dealing with ecology, energy, the environment, climate change, energy efficiency and related issues. The definition of the category set out in the WoS database reads: "This category covers resources that focus on basic and applied research on green and sustainable science and technology, including green chemistry; green nanotechnology; green building; renewable and green materials; sustainable processing and engineering; sustainable policy, management and development; environmental and agricultural sustainability; renewable and sustainable energy; and innovative technologies that reduce or eliminate damage to health and the environment" [16].

An analysis of universities' scientific activity in a given area may be broached from the perspective of their scientific and technological output. Scientometrics and bibliometrics are pivotal to the analysis, measurement and assessment of research activity [17]. Those disciplines frequently deal with the analysis of (researcher, group, institution, discipline or country) research and its impact on the scientific community. They nonetheless also embrace more innovative pursuits, such as the detection of new research fronts and emerging fields, network analysis and the identification of research niches.

In those endeavours, the information sourced from publications and patents may be supplemented by the analysis of research projects. When implemented under competitive research and development programmes, such projects furnish data apt for weighing basic against applied science and assessing the effort deployed in emerging fields or interdisciplinary or cross-border research, such as is often the case in environmental and socio-economic studies [18].

Further to this interest in sustainability, some authors have explored the field qualitatively [19,20]. Others have adopted a bibliometric approach to examine the development of sustainability science through the analysis of citations [21,22], journal interdisciplinarity [23] or social media repercussions [24,25]. Recent papers have analysed the dynamics and evolution of GSST (considering that it is an approach to sustainable research) [25,26] and others have focused on the specific study by research on higher education for sustainable development [27].

Earlier studies have also analysed scientific activity in related areas using European projects as a source of information to identify university activity and to analyse project content in depth [28]. The present authors developed a model based on project publications to relate research activity to the impact on the academic community as reflected in social networks [29].

As some authors mention, given the increasing interest on the sustainability in higher education, there is a clear need for a systematic review of the literature [30]. Then, HEIs around the world have become engaged in sustainable practices and methodologies. This is reflected in the "growing body of literature has investigated this trend" [31]. However, as is mentioned by several authors there are "a lack of studies analyzing impacts from a more holistic perspective" [4]. The measurement of these impacts also involves considering that "the tools generally lack indicators in research, education and community engagement areas" as was explained by authors as Yáñez, Uruburu, Moreno, and Lumbreras [10].

Against this backdrop, this study has focused on the Spanish case. This country was chosen because Spanish universities, through the Conference of Rectors of Spanish Universities (CRUE), show great interest in sustainability issues. In fact, CRUE had created a working group on environmental quality and sustainable development. Therefore, CRUE has had a very active role. As some authors mention "this working group compiles university experiences and the progress made in the area of environmental management and sustainability, promoting cooperation among universities in these areas" [32]. In addition, the Spanish university system is the third country in Europe in terms of the number of STEM graduates [33]. It is among the first in terms of the number of the adult population with a university education [34], and it is a clear reference also for the Ibero-American region. In this sense, according to data from the Ibero-American Network of Science and Technology Indicators, Spain is the number one country by number of doctors [35]. From the point of view of scientific production, its contribution is also very relevant since 75% of the country's publications come from the higher education sector [36]. Furthermore, it is the number one Ibero-American country in terms of both the total number of publications and publications per researcher [35].

Considering this scenario, the objectives pursued here are to:


#### **2. Materials and Methods**

Considering the importance of the analysis of scientific activity on sustainability in the higher education institutions, this work focuses on the study of reports, scientific papers and research projects. In this line, the methodological framework used is the Information Metric Studies, which allow, through bibliometric techniques, the analysis of publications and projects on the proposed topic.

Information for this study was sourced from the following:

	- 'Evaluación de las políticas universitarias de sostenibilidad como facilitadoras para el desarrollo de los campus de excelencia internacional' [assessment of university sustainability policies in furthering the international campus of excellence programme] (2010) and;
	- 'Diagnóstico de la sostenibilidad ambiental en las universidades españolas' [diagnosis of environmental sustainability in Spanish universities] (2017);

The study was broken down into the following stages:


#### **3. Results**

The findings are set out below.

#### *3.1. Sustainability Measurement in Universities*

Integrating sustainability in universities entails creating tools that enable institutions to assess their engagement with the economic, social and environmental dimensions of sustainability and to continually improve their performance in those realms. Measuring sustainability remains a complex and challenging process for higher education institutions, however, especially institutions in the early stages of their sustainable development programmes [37].

The three resources used to analyse and measure sustainability in universities were accounts; narratives such as reports and similar; and indicators [38]. The proliferation in recent decades of papers of the many tools in place attests to the interest in such measurements. Some of the most popular publications analyzing and comparing sustainability assessment tools were published by authors such as Shriberg [39]; Alshuwaikhat and Abubakar [40]; Yarime & Tanaka [41]; Sayed, Kamal and Asmuss [42]; Gómez, Sáez-Navarrete, Lioi, and Marzuca [37]; Salvioni, Franzoni, and Cassano [43]; Berzosa, Bernaldo and Fernández-Sanchez [44]; Findler, Schönherr, Lozano and Stacherl [45] or Parvez and Agrawal [46]. Then, the tools have been described in the international literature to quantify sustainability-related activities in higher education institutions. According to Alghamdi [47], the ones most commonly used include the following:


Some of the most prominent features (dimensions and indicators) of the afore mentioned 12 tools are set out in Table 1. The links and references consulted to extract the information are shown in the table.

These tools differ in typology, the number of indicators and methodology for determining the integration of sustainability in university activities. By way of example of the details for some of these tools, the parameters addressed (and their relative weights) in sustainability tracking, assessment and rating systems (STARS) include academic courses (AC) (28%), engagement (EN) (20%), operations (OP) (35%), planning and administration (PA) (15%), innovation and leadership (IN) (2%). The academic dimension covers indicators associated with research and projects, with three sub-sections: support for research, research and scholarship and open access to research, further broken down into items.

Sustainability assessment questionnaires (SAQs), in turn, comprise eight dimensions: curriculum, research and scholarship, operations, faculty and staff, development and reviews, outreach and services, student opportunities, and administration, mission and planning They are designed to stimulate discussion and further assessment by campus representatives knowledgeable of and responsible for the activities specified and include a specific dimension for assessing research on sustainability.

Green Metrics, another tool, encompasses dimensions such as education (ED) (18%), (setting and infrastructure (SI) (15%), energy and climate change (EC) (21%), waste (WS) (18%), water (WR) (10%) and transportation (TR) (18%). The education dimension covers research and project indicators and more specifically, total research funds (in USD) dedicated to environmental and sustainability research and the number of scholarly papers published on the environment and sustainability.


**Table 1.** Tools for measuring sustainability in universities: Key features.

In short, sustainability measurement in universities embraces a variety of realms, some internal and others external to the university community, as reflected in the diversity of parameters defined in these tools and the number of indicators applied to each.

These 12 tools consider indicators that can be grouped within each of the categories (curricula and competences, research, campus operations, community outreach, university and governance, sustainable assessment and reporting) that have been collected in the model for sustainable development in universities by authors as Alonso-Almeida et al [48] and Lozano et al [49,50]. However, as mentioned above, each tool uses different indicators and weights for each dimension (Table 2).



The research dimension is explicitly included in all tools, except STARTs and GM which include it in academics and education, respectively. However, the sustainability research dimension uses a small number of indicators compared to the total of the indicators of each tool, mainly related to the number of publications, number of projects, funding support, institutional support, staff (teachers and/or students) involved in this type of research. Some examples are: In the GM tool of 33 indicators, there are only two focusing on sustainability research (number of academic publications and funds); and in the SAQ tool that has 35 indicators and only three are related to research (projects and professors focused on this subject, as well as the presence of multidisciplinary or interdisciplinary structures for research on sustainability issues).

#### *3.2. Sustainability in Spanish Universities*

The Spanish university system presently comprises 83 institutions (50 public and 33 private). Together they account for 27% of total national R&D spending and employing 36.8% of the personnel engaging in research [33]. The intensity of their activity is attested to by the findings, for Spanish universities author 75% of the country's scientific publications listed in international databases, such as the Web of Science.

Most Spanish universities conduct some manner of sustainability-related activity (administration, education, research), although they only began to address this issue in the nineteen nineties, somewhat later than countries, such as America and other parts of Europe. Those initial endeavours around university sustainability in Spain were induced by inter-university projects and attendance at meetings or seminars on the subject. Against that backdrop, in 2002 the Conference of Spanish University Vice-Chancellors (Spanish initials, CRUE) created a working group on environmental quality and sustainable development to further pro-sustainability action in Spanish universities. In July 2008, the group was restructured into the Sectoral Commission on Environmental Quality, Sustainable Development and Risk Prevention in Universities (Spanish initials, CADEP) [51]. In 2009, the commission was renamed CRUE-Sustainability. At least 68 universities have participated in some of its events. In June 2007, the University of Santiago de Compostela hosted a standing seminar for university environmental action entitled "Indicators and Sustainability in Universities". The conclusions included a proposal to create a technical working group to establish a system for assessing sustainability performance in Spanish universities. That project envisaged the definition of system of indicators to assess the universities' sustainability policies, compile models for implementing those policies and identify good practice in connection with international campus of excellence projects. The result was a system of indicators for assessing university sustainability agreed to and tested by most of Spain's university system. On an initiative authored by several universities, that led 2 years later to the creation of the CRUE Sectoral Commission-Sustainability to compile the experience of higher education institutions on environmental management, progress in heightening environmental awareness in the university community and risk prevention. It was also intended to further cooperation and exchange in these areas and establish good practice.

Those commissions and working groups organised a series of sustainability-related activities in Spanish universities and authored reports on the subject. In 2010, a questionnaire was circulated among all higher education institutions, to which 30 public (62% of the Spanish University System) and one private university responded. The findings were written up in a report [52] diagnosing university sustainability policy in Spain and defining the realms and indicators to be used to measure Spanish universities' contribution to sustainability. Those indicators afford an assessment framework for the progress made in sustainability policies, rendering progress more visible for the university community and society at large. The 31 universities responding to the questionnaire were consulted about the structure of the report. All were visited to discuss their inquiries and suggestions and obtain a first-hand view of sustainability programmes in the Spanish university system. The report revealed that the universities analysed engaged most intensely in areas such as environmental awareness, waste management and sustainability courses and least in green procurement, water management, social

responsibility and environmental impact assessment. The follow-up on that first report (2010–2011) included the review of the tool used by the CADEP-CRUE working groups and the one employed by the group on university sustainability assessment (GESU) from November 2014 to April 2015. The outcome was a new document entitled "Sistema de evaluación ambiental de la Universidad Española- GESU-CRUE v3", Spanish university environmental assessment system-GESU-CRUE v3.

A second version of the questionnaire formulated in 2017 and responded to by 33 universities spawned the report "Diagnóstico de la sostenibilidad ambiental en las universidades españolas' (diagnosis of environmental sustainability in Spanish universities). One of its major conclusions was that the universities studied had made environmental improvements in terms of organisation, with the highest mean scores reported for environmental policy, awareness and engagement. Nearly all the respondent institutions had environmental policy officers and had attained a degree of engagement deemed as acceptable by the university community. Education and research, areas where implementation was essentially nil, scored the lowest. The findings on green campus management showed that universities had made a substantial effort in controlling environmental parameters such as water, energy, waste and biodiversity, although the implementation of improvements in those areas was scantly systematised. The difficulties were also observed in the adoption of measures to enhance green procurement. Many universities had developed plans for improving sustainable transportation, although most were still in the initial phases [51].

According to the results of the two surveys (Figure 1), the universities listed in the 2017 report scored higher in sustainability policy, water management and environmental impact assessment than the institutions included in the 2010 report. In En relación a los demás aspectos analizados, cabe destacar que una de las fortalezas de las universidades que se recoge en los dos informes es la implicación de la comunidad universitaria, en este ámbito se incluyen: another vein, one of the strengths recorded in both reports was university community engagement, including cursos, jornadas, noticias, congresos, difusión de actividades de la sociedad en general, etc. courses, seminars, press releases, congresses and dissemination of activities to society at large. The 2010 and 2017 reports also revealed considerable room for improvement in connection with environmental impact assessment, social responsibility, green procurement, education and research.

**Figure 1.** Conference of Rectors of Spanish Universities (CRUE) report findings, 2010 and 2017.

The differences between the two reports were not significant due to most of the areas considered maintain very similar results. With the exception of the involvement of the university community, teaching and research which, although they have reduced their scores (even some by 1.8 points) continue to be above the average of the scores. These differences are related to the improvement adjustment of the report, which led to modify some aspects of the questionnaire and simplify the total of questions (from 176 to 140) but keeping most of the questions the same as in the 2010 version so that the information can be comparable. In the case of the social responsibility aspect in the 2017 report, no data is available because it was not considered.

In short, in the results of the 2010 and 2017 reports, Spanish universities stand out in the score of sustainable policies and the involvement of the university community, but they must improve in the other areas that correspond to the effective implementation of sustainability, as well as in the monitoring and evaluation of the environmental impact generated by HEIs.

#### *3.3. Publications on Sustainability*

The Web of Science database lists 79,014 papers in Green and Sustainable Science and Technology, GSST (through 2018). Considering the whole period analysed, the average annual increase was 17.3%.

Although documents dated as early as 1994 were listed, the number of publications was higher in the last 5 years. As can be seen in Figure 2, a 66% of production is concentrated between 2014 and 2018. It is important to note that the decrease in the number of documents in 2018 is due to the updating of the database.

With 20% of the total, China had the highest output, followed by United States, India and England. Whilst the major countries obviously had the highest output, those values were normalised to accommodate size with the activity index (AI), which compares a country's output in a given area (GSST in this case) to its total output in the WoS database. An AI of > 1 is indicative of greater than expected output in a given field. As Table 3 shows, Netherlands, Belgium, Austria, Malaysia and Australia had particularly high AI values in GSST. In contrast, United States, England, Germany, France and Japan published fewer papers on sustainability than expected.

As Table 3 shows, Spain (with 4179 papers) ranked fifth by output, accounting for 5.3% of all world papers and exhibiting the mean yearly growth of 21.8% (3.3 points more than world growth). As shown in Figure 3, Spanish production has grown steadily and 70% is concentrated in the last 5 years. The importance of this production is also evident in the increase in the world's contribution (2% in 1994 and 5.8% in 2018).


**Table 3.** GSST output by countries (> 1% of total) and activity index.

Source: Compiled by authors based on Web of Science information.

**Figure 3.** Number of Spanish GSST papers listed in WoS, 1994–2017. Source: Compiled by authors based on Web of Science information.

Whilst all the papers listed dealt with GSST, the publishing journals were also indexed under other subject categories. According to Figure 4, which depicts Spain's and the world's output by subject area, Spain followed the same pattern as the world mean, with a slightly higher proportion in environment-related disciplines only.

**Figure 4.** Number of papers by topic (Spain and world mean). Source: Compiled by authors based on Web of Science information.

**.** 

Spanish output was concentrated in major cities such as Barcelona, Madrid and Valencia, although some activity was recorded in all 17 of the country's regions. This is logical, given that these cities concentrate a significant proportion of research centers and universities that are the main producers of the papers analyzed. In the distribution shown on the map in Figure 5, the size of the nodes denotes the output volume.

**Figure 5.** Distribution by city of GSST papers authored in Spain. Source: Compiled by authors based on Web of Science information.

Interestingly, Spanish output was highly concentrated in its universities, with 91% of the papers authored by at least one higher education institution.

The National Research Council (CSIC: no breakdown by institution is provided) produced more papers than any other organisation, whilst the most active universities were the Autonomous University of Barcelona and the Technical Universities of Valencia, Madrid and Catalonia (Table 4). As shown in Table 3, polytechnic universities have intensive production in the GSST field (AI > 2), as do other small or medium-sized universities such as La Rioja (AI = 4.1), Lleida (AI = 3.7), Almería (AI = 3.4) and Jaen (AI = 3.1). On the other hand, the large universities, being generalists, present a lower intensity of production than expected.

Spanish universities partnered most intensely with institutions in UK, Italy, USA, Portugal, France and Netherlands. Figure 6 shows the number of documents in co-authorship between Spanish universities and foreign partners (the size of the node is proportional to the number of documents). The main partners are the European and North American leaders in the topic studied (green circles in Figure 6). However, there is not such an intense relationship with large producers from other regions such as China, India, Australia, and South Korea.

**Figure 6.** Countries of origin of Spanish universities' major partners. Source: Compiled by authors based on Web of Science information.

Following an analysis of the content of GSST papers, 12 of them mention tools for measuring sustainability in universities. All their authors were affiliated with universities: five in Canada, two in Saudi Arabia, two in USA and one each in Chile, UK, India and Mexico. The journals publishing the largest number of such papers were: *Journal of Cleaner Production; Sustainability; Environment Development and Sustainability; International Journal of Sustainability in Higher Education*.


**Table 4.** Output by institution (> 2% of documents).

Source: Compiled by authors based on Web of Science information.

#### *3.4. Sustainability Projects*

European calls for projects have become one of the main avenues for scientific and technological activity and a major source of funding for Spanish institutions. For the analysis of projects related to sustainability, the call of the 7th Framework Programme has been chosen and information retrieved from the CORDIS database confirmed Spain's significant participation in this call (Spanish institutions took part in a quarter of the total projects) (Table 5).


**Table 5.** Distribution of projects under Framework Programmes, by call (all topics).

Source: Compiled by authors based on CORDIS information

According to a report by the Centre for Technological Development in Industry, in the Seventh Framework Programme (FP7, 2007–2013), Spain ranked in the same sixth place in funding received (after Germany (17.8%), United Kingdom (17.2%), France (12.5%), Italy (9.3%) and Netherlands (8.4%)) as it had under FP6. Qualitatively speaking, the results attained in FP7 were substantially better than in the preceding edition, where Spain headed 10.7% of the projects, compared to 6.0% in FP6.

The Seventh Framework Programme comprised 23 calls in different areas. Spain participated in over 50% of the projects awarded under eight of those calls. It was particularly active in FP7-SME (Specific Programme-Capacities: Research for the benefit of SMEs), designed to strengthen small and medium-sized enterprises; FP7-KBBE (Specific Programme-Cooperation: Food, Agriculture and Biotechnology); FP7-INFRASTRUCTURES (Specific Programme-Capacities: Research Infrastructures) and PF7-NMP (Specific Programme-Cooperation: Nanosciences, Nanotechnologies, Materials and New Production Technologies).

Six of the subjects addressed by those projects were associated with sustainability. Environmental protection attracted the largest number of projects (823), whilst 264 addressed energy savings. Spain had a significant presence in all six, with participation ranging from 34% in biofuels to 73% in waste management (Figure 7).

Universities were scantly involved in these projects, contributing to 16% at most. The highest rates of participation were observed in projects on sustainable development and environmental protection. The latter was the area where the largest number of higher education institutions participated (Table 6).


**Table 6.** Spanish university participation in FP7 sustainability-related projects.

Source: Compiled by authors based on CORDIS information.

A total of 40 Spanish universities, 39 public and one (Pontifical University of Comillas) private, participated in European sustainability-related projects. Technical universities participated most intensely, with the Technical University of Madrid heading the list. Taken together, however, the Catalonian institutions (Autonomous University of Barcelona, the Technical University of Catalonia and the University of Barcelona) had a significant presence in these calls (Table 7).

**Figure 7.** Total sustainability-related projects and Spanish participation. Source: Compiled by authors based on CORDIS information.

**Table 7.** Universities participating in FP7 university-related projects (>2 projects).


Source: Compiled by authors based on CORDIS information.

#### **4. Discussion**

The assessment mechanisms are useful tools for diagnosing universities' sustainability performance. The most prominent international studies conducted on the subject have as a rule used information delivered by green metrics, a tool that processes data very generically and yields no details on research particulars such as project participation or paper write-up. Using this tool to analyse nine Indian HIEs, Parvez and Agrawal [46] found that only two had made formal sustainability progress in terms of education. The specific indicators studied were academic courses on sustainable development and the existence of a website on sustainability. They did not, however, analyse research findings or papers on sustainability. Similarly, Marrone, Orsini, Asdrubali and Guattari [53] analysed the green metrics scores for universities in several countries and regions (India, Indonesia, Japan, USA, Canada, Africa, Middle East, South America and European Union). The findings were also given generally, by category only and focusing on inter-country comparisons, with no information on indicators. In education, Canada and the European Union scored highest. Suwartha and Sari [54], in turn, studied 25 US universities that used the tool in 2011, analysing the scores for each by indicator, although in this case the authors grouped education under the category, setting and infrastructure.

Beringer, Wright and Malone [55] studied the state of sustainability in higher education (SHE) in Atlantic Canada, in which the tool of choice was SAQ. They found that the majority of higher education institutions in Atlantic Canada were engaged in sustainable development work, most notably in the area of curriculum. Sustainability research and scholarship is spread amongst faculty and students. Many institutions have inter- or multi-disciplinary research structures to address sustainability questions across campus and in collaboration with community partners. They nonetheless acknowledged that the dimension to be still only moderately developed and identified student commitment to research projects as an avenue for speedier progress.

Consernign SAQ, a paper by Lidstone, Wright and Sherren [15] on sustainability research reviewed 21 Canadian HEIs that used this measuring tool. Their findings showed that 50% of these universities' research plans included sustainability goals. The particulars most intensely studied by these universities included funding, interdisciplinarity and structures (organisation and other university resources). This tool was also analysed by Parvez and Agrawal [46] in their study of nine HIEs in India. The findings on research delivered by STARS are somewhat more detailed, for they include three groups of research-related indicators: research and scholarship, support for research and open access to research, with at least 10 items under each. The latter authors concluded that two of the nine Indian universities, in addition to offering courses on sustainability, undertook initiatives to establish new courses and mechanisms to support and fund research.

Likewise using STARS data, Salvioni, Franzoni and Cassano [43] analysed three groups of universities in the 2015 international Academic Ranking of World Universities (ARWU) Top 500: The first twenty best positions in the Top 500, an intermediate group formed by the last twenty universities classified among the Top 100 and the last twenty positions in the Top 500. The authors acknowledged that universities in the first group assumed sustainable culture more effectively in areas such as research. They stressed that "there is higher integration and inclusion of the sustainability theme in the institutional, managerial, research and teaching activities of universities placed in the Top 5 rankings compared to universities in subsequent position".

In another study, Alshuwaikhat, Adenle and Saghir [56] using SAQ, found public universities in Saudi Arabia to be in the initial stage of integrating sustainability in both the curriculum and research, stating "sustainability-related projects are not prioritised within universities and sustainable financial management practices are not significant".

It is important to note that all these sustainability assessment tools are used by universities on a voluntary basis. However, public authorities in most countries are promoting the transparency and accountability of universities through more information about their activities to society, which may include sustainability reports. In the area of public funding of universities, the use of allocation mechanisms at least partially performance-based for teaching and research is spreading [57], but they still do not include sustainability indicators. In many countries, the most important outputs taken into consideration for performance-based funding are those related to research, which also do not introduce criteria that promote sustainability research.

The case of Spain, as mentioned above, is interesting because of the relevant role played by the higher education system. It also has its own initiatives for the analysis and measurement of sustainability in universities. When analyzing the results obtained in this study, it can be observed that in terms of measures for the development of sustainability, universities scored highest in items relating to university community engagement (7.6), waste management (6.5) and campus grounds and biodiversity (6.0) and lowest in environmental impact assessment and green procurement (<5.0). The 2010 and 2017 reports also revealed considerable room for improvement in all areas, especially environmental impact assessment, social responsibility, green procurement, education and research. Although environment-related degrees have been in place in the last two areas, sustainability has yet to be integrated holistically across all university activities, rather than as a separate chapter or stand-alone curricular content.

The Spanish experience has also served as a reference for other regions. This is why a similar study was carried out in Latin America under the title "Definición de indicadores para la evaluación de las políticas de sustentabilidad en universidades latinoamericanas" (definition of indicators for assessing sustainability policies in Latin American universities). According to the findings, the region's countries scored lower than Spain. With a score of 6.1 (higher than Spain's 3.3), social responsibility was the sole exception and the only item attesting to significant commitment to sustainability [58]. In all the other items analysed (sustainability policy, university community engagement, education, research, urban planning, energy, water, transportation, waste management, green procurement, environmental impact assessment), Latin American universities scored lower than their Spanish counterparts, in particular in connection with green procurement and transportation. The report consequently identified a pressing need to develop and integrate sustainability in all aspects of university life.

Spanish governing bodies introduced a new way to allocate public universities funding based on performance criteria. By funding universities according to their outputs, rather than inputs, state policy makers in Spain believe they are providing an incentive for universities to improve their quality management and accountability [59]. In other words, it is about universities optimizing their management and activities considering ethical values and transparency to achieve a greater positive impact on all aspects of society. One of the main outcomes of this performance funding system has been the greater number of strategic plans or sustainability reports articulated and published online by Spanish universities in recent years [60].

Reviewing R&D+I from a bibliometric perspective, the number of scientific papers constitutes a good measure of the scientific community's interest in a given subject. The creation in Journal Citation Reports of the subject category Green and Sustainable Science and Technology (GSST) laid the ground for measuring the impact of sustainability-related research.

Although the earliest papers on the subject date back to the nineteen nineties, the number of articles has risen substantially in the last 5 years, especially at a yearly rate of over 17%.

Whilst research majors (China, USA, India, etc.) account for the highest output in GSST, smaller countries (Netherlands, Belgium, Austria and Malaysia) devote more effort to the area relative to the total research effort deployed.

Spain ranks fifth worldwide by total number of articles on the subject and its output is growing faster than the world average (21.8% versus 17.0%). The intensity of the country's activity in the area is also attested to by its activity index which, at 1.8, denotes greater devotion to the area than expected on the grounds of its overall WoS-listed output. Further WoS evidence for that assertion lies in Spain's ninth place in overall output compared to its fifth place in papers relating to sustainability.

Another feature of Spain's activity in the field is the concentration of sustainability research in three of its major cities (Barcelona, Madrid and Valencia), cities that concentrate the great universities. The presence of powerful university systems in all three is consistent with the fact that universities account for 91% of Spain's output in this subject category and with the community's interest in sustainability and related issues.

Although the country's most productive institution is the National Research Council, four universities also lay claim to significant output figures: The Autonomous University of Barcelona and the Technical Universities of Valencia, Madrid and Catalonia. Together, those five institutions account for over 20% of Spain's entire production in this area of research. Likewise, it has been shown how some small and medium sized universities present an intense productive activity in subjects related to sustainability.

Spain also plays a significant role in European projects on the subject, with a success rate of over 30% in the most prominent sustainability-related calls organised under the Seventh Framework Programme. One less favourable aspect of that success is that universities participate only marginally in such calls, where the private sector prevails. Earlier studies have shown that this is not uncommon in several other areas [21]. In any event, the same four universities found to reign in scientific paper output also rank highest by the number of Seventh Framework Programme projects, accounting for over 30% of the projects awarded in the area to the SUS as a whole.

Spanish universities' scant presence in sustainability projects should prompt academic management to change their research strategies for the present findings confirm the HEI community's interest in engaging in science on the subject. One possible way to raise participation would be through partnering with the private sector, for companies have acquired an ever more significant role, comparable to their predominance in patent applications in areas such as renewable energies.

It is evident that the use of scientometric tools, such as the analysis of publications or projects, has certain limitations. Among them is the difficulty of accurately defining the specific area of sustainability, so all quantitative studies are an approximation. However, the use of tools external to the information provided by the universities themselves can contribute to reducing the biases of interpretation (or manipulation) of the data. Likewise, the use of absolute indicators, such as the number of projects or publications, combined with relational indicators, such as the activity index, can offer a good measure of the effort of each of the institutions. In this sense, the case of Spain shows that the limitation raised in the literature on the scarcity of indicators on research in universities (compared to the total of the indicators of each tool) can be approached in a complementary way from a scientometric perspective.

#### **5. Conclusions**

After the development of this study, some relevant aspects should be mentioned as conclusions. It has been detected that the interest in sustainability has been growing and this is evident in publications and projects on the subject as well as in the initiatives developed in different environments to promote it. The higher education sector is presented as one of the most fertile spaces for the development of measures in this field.

The methodology used in this study, which combines bibliometric techniques with the analysis of institutional documents, may be useful to identify a country's commitment to the development of initiatives, such as those related to sustainability.

The Spanish case can be taken as a reference to analyze the situation of other European and Latin American countries and detect points and strengths with respect to the implementation of measures on sustainability in universities.

The most common tools for measuring sustainability in universities use a small number of generic indicators (compared to the total of the indicators of each tool) of scientific output on sustainability. The presented study of the Spanish case shows the possibility of including several more specific indicators to analyse the topic of university research in sustainability (thematic specialization and AI). It would be very interesting for the different tools to include some of these indicators collected externally with an objective criterion, which would allow studies to be carried out with more reliable data in order to make international comparisons and facilitate the university's accountability in this area.

Regarding public policies that aim to promote the sustainability of universities, it should be noted that although the use of the tools studied is voluntary for the institutions, actions in terms of transparency and accountability can help to promote measures that encourage information on the impact of their actions on society. In the field of research, the incorporation of indicators on sustainability research, analysed for the Spanish case, in the performance-based funding models would provide an important incentive to promote research in this field.

**Author Contributions:** Conceptualisation, all authors; methodology, D.D.F.; data cleansing and formal analysis, D.D.F. and L.A.S.-H.; writing—original draft preparation, D.D.F. and L.A.S.-H.; writing—review and editing, F.C. and E.S.-C.; project administration and fund raising, D.D.F. and E.S.-C.

**Funding:** This project received funding from the European Union's Horizon 2020 Research and Innovation Programme under grant 741657, SciShops.eu. The content of this article does not reflect the official opinion of the European Union. Responsibility for the information and views expressed in the article lies entirely with the authors.

**Acknowledgments:** The methods used were developed on the framework of the project entitled 'Detection of new research and innovation fronts. Analysis of knowledge flows in the scientific domain, industry and society in the field of energy efficiency' (ref.: CSO2014-51916-C2-1-R), funded by the Spanish Ministry of the Economy and Competitiveness.

**Conflicts of Interest:** The authors declare no conflicts of interest.

#### **References**


© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
