**The Motivation of Students at Universities as a Prerequisite of the Education's Sustainability within the Business Value Generation Context**

**Alzbeta Kucharcikova \*, Martin Miciak, Eva Malichova , Maria Durisova and Emese Tokarcikova**

Department of Macro and Microeconomics, Faculty of Management Science and Informatics, University of Zilina, Univerzitna 8215/1, 01026 Zilina, Slovakia; Martin.Miciak@fri.uniza.sk (M.M.); Eva.Malichova@fri.uniza.sk (E.M.); Maria.Durisova@fri.uniza.sk (M.D.); Emese.Tokarcikova@fri.uniza.sk (E.T.) **\*** Correspondence: Alzbeta.Kucharcikova@fri.uniza.sk; Tel.: +421-41-513-4426

Received: 16 September 2019; Accepted: 8 October 2019; Published: 10 October 2019

**Abstract:** The aim of this article is to identify substantial factors affecting the motivation of universities' students to be actively engaged in the education process and define recommendations for the increase of this motivation. As a result, the sustainability of education at universities will be supported, contributing to the increase of the value of human capital of students and, subsequently, to the generation of value for the stakeholder groups in those enterprises where the graduates will be employed. The research hypothesis is focused on the presence of differences in students' motivation in relation to their gender, study program, and the year of study. To effectively achieve this aim, the analysis, comparison, and the synthesis of the theoretical background was performed, using available sources of secondary data found in the pieces of domestic and foreign professional literature. The pieces of knowledge obtained were supplemented and combined with pieces of information acquired from the questionnaire survey conducted, focusing on the motivation of students of informatics and management at a university in the Slovak Republic. As tools of statistical analysis, tests of independence suitable for nominal categorical data were applied. It was revealed that young people are motivated to study at a university, specifically at the Faculty of Management Science and Informatics, mainly by the prospect of better chances in the labor market, the possibility of getting a higher salary, and higher qualification. The motivation to study at a university in order to improve the opportunity of getting employed in the labor market was more frequently perceived by women. Despite the fact that the level of teaching is considered to be high by almost 50% of the students regardless of their gender, study program, or the year of study, their motivation also stems from their expectations related to their future jobs. The students of informatics expect to have a team of friendly colleagues, delightful and stimulating working conditions, and the opportunity to do meaningful work. Among the students of management, meaningful work was replaced by the opportunity for self-fulfillment. When focusing on other factors, the differences based on the gender, study program, or the year of study were not statistically significant. Based on these findings, specific measures for the faculty's management were proposed.

**Keywords:** motivation; human capital investments; generation of business value; sustainability; universities

#### **1. Introduction**

The current time is characterized by open markets and rapid changes in production and sales conditions. At the same time, the risk of new economic crises emerging is still rising. To strengthen the competitiveness of whole economies but also of the individual enterprises, it is necessary to increase the performance and efficiency, which requires the implementation of new technologies and changes in the business processes, the orientation on value management, education, and the emphasis put on the increase of the value of human capital because people represent the most important driver in enterprises and in society.

Human capital includes all the innate and during-life-acquired knowledge, skills, experience, and talent of a person [1].

The scope, structure, and focus of the investment in human capital are affected by macroeconomic as well as microeconomic factors. These include, for example, the society-wide or an enterprise's strategic goals [2], current state within the industry and the enterprise's position in the given industry [3–5], corporate culture [6], the enterprise's orientation on social responsibility [7,8], or the support of sustainability in the long term [9].

Based on the research conducted earlier [10], it was revealed that the enterprises as well as society consider the investment in education to be the most familiar form of investment in human capital. This investment serves for the increase of the level of knowledge and skills of individuals, and for the desirable change in their attitudes.

Education is provided by various institutions. These also include universities [11]. For universities to be able to provide high-quality education for the needs of the practice, they need to identify and affect the factors that motivate young people to enroll for a specific university program and to study. To secure the sustainability of education at universities [12] as well as within the implementation of the value management in enterprises [13], it is also necessary to know the wishes and expectations of students [14] related to their employment in the labor market [15], or more specifically, within the practical operation of enterprises. Accordingly, universities can plan and implement adequate measures afterwards [16]. While increasing the value of human capital via university education [17], the success achieved is influenced by multiple factors. These encompass the phase of the economic cycle, historical development, and the engagement of the country in international structures, measures of economic policy [18,19], attitude of society towards education [20], demand for the highly-qualified workforce on the side of enterprises and the overall situation in the labor market, the level of science and technology in the country [21], quality and reputation of a particular institution providing the education, quality and attractiveness of the study programs and their alignment with the current needs in the labor market, quality and attitude of the teachers to the education process, teaching methods being applied, motivation of students, etc.

A pivotal element in the process of education is represented by the students themselves. There are numerous factors affecting students during their studies, but the most important one is their motivation for studying and for becoming proficient experts within specific needs of the practice. The motivation for studying is a prerequisite for successful achievement of expected results within the education process as well as for the sustainability of this process.

This will also contribute to the increase in value generation in enterprises that employ successful graduates with high level of their human capital.

This was the reason why this particular research was focused on the identification of substantial factors that affect the motivation of students to start studying at a university and actively participate in the education process as well as their expectations related to their future jobs.

Via the identification of these factors of motivation together with the designed recommendations, this research will contribute to the solving of the issue of securing the education's sustainability (which contributes to the increase in the value of people's human capital), while respecting the requirement of enterprises for the increase in the generation of value for the stakeholder groups within the implementation of the value management concept.

The logical connection of the above-mentioned professional and scientific areas is depicted in Figure 1, and it is more thoroughly elaborated in the following sections. However, this is a typical situation happening in research when the real system is quite complex and consists of an abundance of elements. One possible solution to coping with this issue is the application of abstraction. Following this method, the research presented in this article works only with selected aspects within the motivational readiness of university students. Subsequently, the research tries to describe the link between motivational readiness (motivation for studying) of students and the sustainability of the education system. Finally, the sustainability of the education system is connected to the increase in the human capital value of university graduates. After being employed in various enterprises, these graduates, with enhanced levels of their human capital, become valuable contributors to the value creation in those enterprises.

**Figure 1.** The logical structure of the studied topic, consisting of individual elements and their relationships.

#### **2. Literature Review**

#### *2.1. Education and the Generation of Value*

In the time of globalization and dynamic changes, education is becoming the decisive force of successful enterprises. More than ever before, it is necessary to maintain the knowledge and skills of employees at a desirable level.

The stakeholder groups of an enterprise are represented by persons, groups, or organizations that are able to affect the enterprise's existence, or that can be affected by the enterprise's activities. Such groups include the owners, creditors, employees, suppliers, customers, competitors, the government, local authorities, non-governmental organizations and pressure groups, communities, and the media. The stakeholder groups defined this way are related to the value orientation of the business. A common goal of all stakeholder groups is the long-term operation and prosperity of the enterprise. Therefore, the aspect of sustainability is important here as well. Employees, as one of the stakeholder groups, embody the carriers of human capital. Via the generation of value for the employees, the enterprise's human capital is growing. This can be determined as the intangible assets [22]. Additionally, the intangible assets represent a factor contributing to the increase of quality of internal processes, which subsequently generate the value. Other assets (machines, equipment, hardware, software, patents, copyrights, trademarks) are the sources of value generation via human capital that is in possession of the employees. Without employees and their knowledge, skills, experience, abilities, creativity, and personal traits, other assets would only be static, idle things or rights.

The generation of value for the employees, which results in the increase of human capital, is closely related to the concept of value management. This is a style of management focused especially on people, acquisition of skills, and support of synergy and innovation with the aim to maximize the total

performance of an organization [23]. Value management is a process consisting of managerial activities performed via approaches and methods used at various managerial levels of enterprises, focusing on the value.

When the principles, methods, and activities of the value management are applied, the value is being generated for the stakeholder groups, which results in their strengthened loyalty to the enterprise—an owner is willing to invest additional capital, a customer becomes a long-term customer, an employee directs his/her activities towards the fulfillment of the goals set, other stakeholder groups spread the enterprise's good name, etc. [24]. According to Obeng, it is necessary to manage relationships with the stakeholder groups, while the strategy can be focused on all or only on a few selected groups. The author uses the concept of the stakeholder concentration index [25].

The procedure of generation of value for employees starts with the expression of this value, continues with its measurement and its transformation into financial and non-financial indicators, and it ends with the determination of activities for the generation of this value. The value for employees can be expressed as an adequate basic salary, a complex system of additional benefits, job security, respecting of payday deadlines, career development and satisfaction of the needs for self-fulfillment, and so on. The measurement of this value can be expressed via the salary, bonuses, an average length of employment according to categories of employees, the number of official praises, the number of opinions expressed, and the number of proposals. The managerial activities generating the value for employees can include the offer of educational and training activities (ranging from practical training to advanced managerial courses) [26], communication of the employees' representatives with the management, increasing of the quality of working environment and positive influence on motivation, performance, and the general interest in employees within the enterprise [27]. The increase in the value is a result of an optimal combination of numerous activities and factors that need to be monitored, analyzed and incorporated in the designing of variants for the solution with the subsequent selection of the optimal one [28].

Another specific feature of human capital in enterprises is the fact that on the one side it represents a part of the enterprise's market value [29], and on the other side, it is closely connected to one of the stakeholder groups within the concept of value management, and thus the employees. The employed people expect that a rich portfolio of motivational tools will be provided for them [30–33], including, for example, meaningful work assignments, an opportunity for self-fulfillment, professional development, fair remuneration, delightful surroundings, job security, friendly atmosphere at the workplace, career advances, fringe benefits, professional management, and others. Via the fulfillment of these expectations, the value is generated for the employees, which develops the human capital available.

Among the internal stakeholders, the employees have considerable importance. Employees primarily acquire knowledge and skills from the education system. The education shall be focused on the creativity of students, independent problem solving, integration of pieces of knowledge obtained from multiple subjects studied and from multiple scientific fields [34]. Within the education process, the cooperation of three elements is necessary: The teacher, the student, and the content of the syllabus.

In the process of education, a university teacher [35,36] is a mediator of the content, who helps a student embrace the knowledge, using the selected, adequate teaching method to familiarize the student with new pieces of knowledge via the application of them in examples. The teacher also points out the common features and relationships within the content and explains the logical structure and connections of individual concepts being taught to the student. The student embraces the pieces of knowledge via his/her active approach to studying. The degree of his/her active approach is directly proportional to the teaching method the teacher selected, which considerably affects the success of the goals' fulfillment. The way of the teacher's activity influences the way of the student's activity in the process of embracing the knowledge.

Subsequently, in enterprises, the value is generated only by those employees (once being the students themselves) who

• participate in continuous education,


The education process shall react to the continuous connection of education and the employment of the graduates in the practice. The graduates will add value to the pieces of information via their interpretation and identification of connections. A precondition for potentially smart employees, managers, or owners of enterprises is represented by well-educated students. These are able to perform a survey or research, seek and select the electronically processed pieces of information needed, and synthesize them into coherent pieces of knowledge. The emphasis is put on the building of the ability of students to get oriented in the plethora of new pieces of information and the ability to utilize them.

An effective way of teaching uses progressive didactic methods and cooperative forms of work. The primary goal is to actively engage the student in the cognitive process. This is based on the cognition via activity, an active relationship of the student with the natural, economic, or social environment in which the issue is being solved [37]. This approach is oriented on the experience gained by the student, and it develops the mental structures in connection with the corresponding processes. In the case of conventional teaching methods, the student is being presented with particular, mutually isolated pieces of knowledge. The most emphasis shall be put on the creativity of students and their own solutions of the assignments, on the integration of the knowledge, and experience from multiple subjects and from real life during the solving of a particular issue. The students need to learn the abilities of self-presentation, effective communication, and self-sufficiency. Subsequently, this will be reflected in their willingness and flexibility for the solving of various problematic situations [38].

In relation to the generation of value for employees as one of the stakeholder groups, it is meaningful for the enterprise to establish cooperation with universities. The alignment and connection of the needs in the practice with the education at universities lowers the need of an employer to organize other forms of education within the enterprise, both with the general or specialized focus. The enterprise's internal education needs to be evaluated together with the strategy of employees' engagement and with the Critical Success Factors [39].

Within the concept of value management, in concordance with the theory of human capital, it is important to focus the generation of value for employees on the elimination of unwillingness of people to share the knowledge and undergo changes in their routine procedures and methods. Education significantly contributes to this. This is the case of education in enterprises but mainly of education at universities, before an individual becomes an employee.

#### *2.2. Education and Sustainability*

Numerous research studies were performed focusing on the sustainability in education and on the education for the continuous sustainable development at universities [40,41]. This research is focused on the identification of expectations and motivational factors of students at a university, specifically on the factors that can influence their decision-making on enrolling, successful studying, and finishing the studies at a university. This will support the sustainability of education at universities and contribute to the increase in the quality and value of human capital that is in possession of the graduates. When the graduates become employed, the research's results can also contribute to the increase of value of enterprises within the concept of value management.

The basic and conventional mission of the university education process is to perform professional preparation and form human capital via the connection between the education process, research, and the requirements of the labor market. Education plays a key role in the development of a personality and engagement of individuals (as a stakeholder group), and it shall not only support the increase in general and specific theoretical knowledge and the knowledge of the world but also the

effective transition of the knowledge to the surroundings and the practical application in personal and professional life. This is an important part of the university's mission since the knowledge and skills of the graduates, based on which they perform their decisions, will be able to considerably influence the quality of life of future generations from the ecological, environmental, social, cultural, economic, and personal perspectives.

The level and quality of education contributes to the sustainable development of the economy at the international as well as at the national, regional, and local level. The results of research studies point out the positive economic effect and the ability to secure sustainable development in those cities and regions in which there are universities and their graduates [42]. In addition to that, the mutual cooperation of universities with enterprises results in various innovation activities in the form of applied new technologies and patents [43]. In a synergic way, the education emphasizing the practical side of the application of the students' knowledge in the business practice increases their professional and technical competencies, cognitive and non-cognitive abilities, which enables them to get decent and adequately remunerated jobs.

Sustainable development is a broad concept that has many forms of application and that is related to the sustainability of the university education itself. For the education to be the driver of development and prosperity of economies, it is necessary to meet these conditions:


Within the economy's cycle, education of high quality, based on the principles of sustainable development, has a considerable impact on the increase of the quality of human capital, which enables the strengthening of sustainable development of enterprises and increases their value and economic prosperity of the whole country. On the other hand, this enables sustainable investment with a substantial impact in the field of university education, contributing to the sustainable development of the universities themselves.

For enterprises and universities/faculties to be able to meet the requirement of the education's sustainability and of the increase in the generation of value for the stakeholder groups, they need to identify and subsequently influence the factors that motivate students to study. This motivation represented the topic on which the questionnaire survey for the students of the selected faculty was oriented.

#### *2.3. Motivation for Studying*

A motive represents an internal reason, which causes a change in the person's behavior and leads to the fulfillment of his/her needs. The motivation for studying or for learning can be perceived in two perspectives. One is created by the motivation of the participants of particular educational activities being performed, which follow the content and the structure of the study program (internal motivation). The second perspective is represented by the motivation for the learning itself, which is affected by the expected benefits from the education (external motivation).

The key element within the realization of the education process is represented by the students themselves. Their motivational readiness for studying depends, among other things, on their emotional state, cultural and educational background, or the physical conditions in the classrooms.

There is an abundance of motivational factors. Within this research, they were divided into three groups:


These factors are closely related to each other and they are in a relationship of mutual influence. Therefore, the strength of motivation is the intersection of all these factors (Figure 2).

**Figure 2.** The factors of students' motivation at universities.

Revealing which specific elements create these three groups of motivational factors will help solve the issue of securing the education's sustainability, respecting the requirement of enterprises for the increase in the generation of value for the stakeholder groups. This can be considered to be an important contribution of this article to application of the research results into practice.

Motivation is the most important factor of university education since the studying at universities is beyond the compulsory school attendance, and thus the motivation of a university student is primarily dependent on the student's personality. In Slovakia, there is also a still present pressure on the successful completion of this type of education from the family members of a student. However, there is a difference between the motivation for studying (learning) and the motivation to only obtain the diploma or the university degree itself. In the first case, the student realizes the desirability and usefulness of the knowledge for his/her future employment and for the personal life, he/she has a strong self-motivation for the advances and cognition. In the second case, the motivation is based on the misconception that a "piece of paper" without the knowledge obtained at a university will automatically help him/her find a job in the labor market. Alternatively, it is based on the assumption that the status of a university student helps postpone the start of the working life. (In Slovakia, even though many students work part-time or have temporary jobs during the studies, the parents still tend to support them financially until they finish their studies.) This perspective can be considered mistaken, leading to incompetence in future jobs. The universities try to correct it via asking for feedback from enterprises and they try to flexibly adapt the content and scope of accredited mandatory and optional subjects.

In general, it can be assumed that the internal motivation of a university student includes the usefulness of the knowledge gained for life, obtaining the qualification for future jobs, curiosity, desire to learn something new, getting the social status of an educated person, and the value system of the student. When talking about the internal motivation, the most frequently used terms shall be the need of cognition, self-fulfillment, self-transcendence, but, at present, also the terms such as the pro-sociality and the job/mission, and the need for life-long learning.

The external motivation can encompass the expectations placed on the student, i.e., the stimuli from the labor market or the demands of employers [58], expected future salary in the given field, professional status and working conditions, social status in an explicit way, i.e., the society's perception of an educated person, the influence of the family and surroundings, or the influence of teachers at lower levels of the education system, prestige of a university, the overall environment at the university (including supportive teachers), subjects with appropriate content and purpose, motivational scholarships or job offers for the best students, lectures done by experts from the practice, internships in enterprises, job fairs, a possibility to participate in international student mobility programs, the need to be in a young and inspiring environment, and the interpersonal relationships [59,60].

The development of information-communication technology (ICT) enabled the increase of the students' motivation as well. It offers various new tools supplementing and supporting the educational process that facilitates studying and increases student engagement [61,62]. The experts then point out the fact that many students, regardless of the chosen direction of their studies, seek studying subjects that can increase their business skills and education [63,64], while these pieces of knowledge help them find better employment in the labor market or start their own businesses. The expectations of a future career are one of the motivating factors for young people to study at a university. By employing highly motivated students, enterprises will attract employees with a high level of human capital. This will also contribute to increasing business value generation for the stakeholders.

The students' motivation for studying at a particular university and their expectations about future careers was addressed in the questions asked within the questionnaire survey focused on the students.

#### **3. Materials and Methods**

The aim of the article is to identify substantial factors influencing the motivation of students at universities to actively engage in the education process and define the recommendations for the increase of this motivation so that the sustainability of this form of education is supported in the long term. This will contribute to the increase of the value of human capital of students, and, subsequently, also to the generation of value for the stakeholder groups in the enterprises in which the graduates of universities will be employed.

The effective attainment of this goal starts with the analysis, comparison, and synthesis of the theoretical background of the studied issues, based on the available pieces of secondary data found in the domestic and foreign professional literature. The studied topic overlaps several theoretical concepts and approaches. The central element is represented by the process of education (specifically university education) as a specific form of investment in human capital. Such investment increases the value of the human capital available and prolongs the period during which it is usable, as it is described in the concept of human capital management. Another perspective is added by the value management concept where education directly affects the value that the enterprise can create for the stakeholder groups. Within this concept, the employees themselves are one of the stakeholder groups. Since students will become employees in the future, it is efficient to deal with the sustainability of their motivation already at the phase of their university studies. The logic behind the theoretical background of this topic together with the interrelationships of individual elements are depicted in Figure 1.

Another step leading to the attainment of the aim is the collection of the primary data. The data points were collected via the method of sociological inquiry with the application of the questionnaire technique. The questionnaire was anonymous, focused on the students of informatics and management of the University of Zilina, Faculty of Management Science and Informatics in the Slovak Republic.

The survey was conducted in 2018, including students from the first and the second grade at the Faculty of Management Science and Informatics. The population consisted of 577 students. A total of 306 filled questionnaires were collected. This sample size put the margin of error to 3.84% at the confidence level of 99%. This survey represents a starting/preparatory exploration within the conditions of the faculty. In the future, it is planned to broaden the scope of the research, including more students at the faculty and at the partnered faculties in Slovakia and abroad.

The purpose of the inquiry was to reveal the current state of the motivation of students for studying at university and of their expectations about their future careers. The identification of students' expectations in relation to their employment in the labor market contributes to the creation of the situation where education at a university helps students obtain the qualification needed for getting jobs that will meet their wishes. The identification features of respondents included the gender, study program, and the year of study. Based on these features, the basic description of the research sample was elaborated. In the questionnaire, three questions were used in which multiple choices could be selected by a respondent. In nine questions, only one choice could be selected. Within the total number of 12 questions included in the questionnaire, in four of them the respondents could also freely express/add their own opinions. Particular questions in the questionnaire were connected to three basic groups of factors affecting the motivation of students (Figure 2). Specifically, the questions were focused on the reasons leading young people to start studying at a university (expectations before starting the studies), the perceived level of teaching (realization of teaching activities), and the expectations related to their future careers. These elements represent basic variables to be used to confirm or reject the main hypothesis underlying the whole research. The content of questions in the questionnaire follows the findings from the literature review and it is also inspired by a previous study conducted in the Czech Republic [65]. Therefore, the validity of the specific questionnaire applied in this research is supported by the application of a similarly constructed tool used in the above-mentioned study. Overall validity was enhanced by the fact that the questionnaire and its final form was checked by several experts within the field of human capital and higher education. To make sure that the tool applied in the research was inherently consistent, Cronbach's alpha was calculated. The result value of 0.73 shows that the tool's consistency is acceptable. However, for future research projects, the tool can be altered to achieve an even higher value.

The research hypothesis was defined as follows: The motivation of students for studying differs in relation to their gender, study program, or the year of study. Here, the motivation consists of three elements included in the aforementioned variables. This way the research hypothesis indicates its decomposition into particular statistical hypotheses (e.g., the reasons leading young people to start studying at a university differ in relation to their gender). In the article, the individual sections of the results part are structured accordingly, with the questions in the questionnaire encompassing the variables applied in the hypotheses' testing. The assessment of the hypotheses is not explicitly listed, only the test results and their interpretation are included in the article.

Within the processing and interpretation of the results obtained from the primary data, specific forms and techniques of exploratory analysis were applied. The processed data outputs were appropriately listed in tables and depicted via histograms for the support of interpretation of the results achieved. The nature of the data entries themselves (categorical data) determined the application of relevant methods of statistical analysis. Depending on the number of categories, the statistically significant differences were detected via suitable methods for the testing of independence of two variables, including the Pearson's chi-squared test and the z-score related to the group of chi-squared methods, at the significance level of α = 0.05. The Pearson's chi-squared test was applied in cases when one of the variables had more than two categories. This test identifies the presence of statistically significant differences based on the comparison of the actually observed and expected frequencies [66,67]. This method of statistical analysis is often used by researchers in various fields, for example, in the field of management, marketing, and business [68–73]. The method is also used in research projects focused on the field of education [74–77]. In cases of the questions in which the respondents could choose more than one answer, the responses were evaluated separately. This way, multiple association tables were created, encompassing the frequencies for dichotomic nominal variables. From these, z-scores were calculated to be applied in the statistical testing of hypotheses. Other research works, including those focused on the field of education, often utilize other statistical methods as well [78–82], but their application is based on working with the numeric data type.

Based on the results of the statistical analysis in combination with the findings extracted from the review of the professional literature, finally, the recommendations leading to the increase of the motivation of students for an active participation in the process of education were defined and described in the article, within the context of sustainability of the whole system.

#### **4. Results**

The results obtained from the processing of the primary data points contribute to the identification and description of the current state of the motivational readiness of students of universities for studying. High motivational readiness is a fundamental precondition for the achievement of high-quality education because it is interconnected with the effort of students to reach great studying results during their studies. Students who actively participate in the education process become qualified job seekers in the labor market after finishing their studies, they have high values of their human capital, and they subsequently become valuable employees of enterprises, contributing to the value generation in those enterprises. Only if these three phases are connected and sufficiently aligned, the sustainability of the education system in the long run can be achieved.

A basic outlook on the processed data can be gained from the description of the research sample captured in Table 1. In terms of gender, men had the majority position within the sample. In terms of the study program, the majority position was held by the students of informatics. In terms of the last identification feature represented by the year of study, the students from the first year of university studies prevailed in the research sample. Even though the proportions of individual groups in the sample are not equal, the numbers of respondents within these groups enable the application of methods of statistical analysis for the detection of differences between these groups (*n* > 20).


**Table 1.** Characteristics of the research sample.

The results obtained from the survey are structured in accordance with three basic groups of factors affecting motivational readiness of students for studying. The first group of factors includes the reasons of students for starting to study at a university. The second group is connected to the realization of educational activities during the studies at a university. The last group of factors gives an account of the students' expectations in relation to their future work positions and assignments.

#### *4.1. Reasons Leading to Studying at a University*

The reasons causing students to start studying at a university represent the elemental source of their motivation during the whole studies. The aggregated results of the corresponding question from the questionnaire are captured in Figure 3 in a graphical way to enhance their interpretation.

**Figure 3.** Reasons leading to studying at a university.

The results show that the two most frequent sources of motivation for starting to study at a university are the effort for increasing one's chances in the labor market (75.16% of respondents) and the opportunity to get a higher salary in the future (69.93% of respondents). An interesting fact is that almost half the students chose the reason of only trying to get the university degree itself for the sake of having it. Contrary to the effort for increasing one's qualification, the desire to only acquire the university degree represents only a superficial interest in the studies themselves, which implies less attention paid to the content of the subjects being taught and lower interest in the active participation in the education process. The opportunity of getting a higher position was chosen only by 15.69% of respondents. This result implies that students do not fully realize that due to the university education they will be able to achieve higher job positions later in their careers. This finding thus creates space for future improvement. Implementation of appropriately directed recommendations for the process of university education can increase the perception of this advantage by the students, which will contribute to higher motivation for achieving superb study results.

Then, within the reasons leading to studying at a university, the presence of statistically significant differences based on the student's gender was analyzed. The purpose was to reveal whether there occurs significant differentness of motives for studying at a university between men and women. Table 2 shows the results of a statistical analysis based on the testing of independence using the z-score together with the *p*-value (at the significance level of α = 0.05) enabling the interpretation of these results.


**Table 2.** Reasons leading to studying at a university in relation to gender.

A statistically significant difference was corroborated by the test only for the motive of the effort for increasing one's chances in the labor market. Women chose this motive more often (86% of women) in comparison with men (69.90% of men). Overall, the motives of women and men were not considerably different. However, the significant difference was identified for the most frequent motive. Since we consider this motive to be a strong factor affecting the effort for achieving great study results, there is space for directing the recommendations toward the increase of perception of this form of motivation among the male students. One of the possible reasons for the difference identified is the socially-conditioned state when women can perceive their chances in the labor market as lower than those of men, and they try to get an advantage via higher education.

Then the attention was paid to the identification of significant differences in the perception of the motives based on the study program. The results of this analysis were reached via the calculation of the z-score again, and they are listed in Table 3 in a structured way, together with the corresponding *p*-values (at the significance level of α = 0.05).


**Table 3.** Reasons leading to studying at a university in relation to the study program.

Within the research sample, there were 79.13% of male respondents studying the study program informatics, and there were 70% of women studying the study program management. Therefore, it is an interesting finding that the results of identification of differences based on the study program as the distinguishing feature did not copy the previous results obtained from the identification of differences based on the student's gender. This is supported by the fact that, in this case, the difference was not detected in the perception of the motive represented by the effort for the increase in one's chances in the labor market. The only statistically significant difference identified from the perspective of the study program was the opportunity for getting a higher job position later in the career. This motive is more often perceived by the students of management (23.01% of students of management in comparison with 11.40% of students of informatics). Such situation can actually be expected since the graduates in informatics aspire mainly to get the positions of programmers and they want to become experts in their professional field. On the other hand, the graduates in management as a study program are expected to get higher up the career ladder to the managerial positions in enterprises after some time.

The last identification feature, whose effect on the results obtained was tested via the techniques of statistical analysis, was the year of study. The purpose of this focus was the evaluation of changes in the perceived motives over time, as a result of maturing, or as a consequence of realized educational activities. In this case, the z-score was calculated once again, following the dichotomic nominal data type. The test's results with the corresponding *p*-values (at the significance level of α = 0.05) and with the relative frequencies of the responses for the individual years of studies are listed in Table 4.


**Table 4.** Reasons leading to studying at a university in relation to the year of study.

In the case of differences based on the year of study, studying the research sample, similar results were obtained to those reached when focusing on the differences in relation to the study program. This can be connected to a more detailed structure of the sample itself. The respondents from the first year of study were mainly from the study program informatics (69.63%), and, on the contrary, the students from the second year of study were mainly the students of the study program management (86.11%). Such structure of the research sample skewed the explanatory power of the differences related to the year of study.

When summarizing the partial findings within the reasons for starting to study at a university, the following conclusions were drawn that will help direct the recommendations focused on the support of the students' motivation with the aim to strengthen the sustainability of the system in the long run. The most frequent motive for starting to study at a university is the effort for the increase in one's chances in the labor market, which can be considered a very positive motive. However, within this motive, a significant difference was detected between men and women. Overall, it was revealed that the study program has a negligible effect on the reasons for starting to study at a university.

#### *4.2. The Level of Teaching at a University*

In the questionnaire survey conducted, the group of factors focused on the quality of the realization of educational activities during study at a university was represented by the question about perceived level of teaching. The aggregate results of this question are listed in Table 5.


**Table 5.** The perceived level of teaching at a university.

Within the research sample, 54.75% of respondents considered the level of teaching to be high and 44.59% of respondents considered it to be medium. This result can be perceived as generally positive, but there is still considerable space for future improvement. Since the realized educational activities during the studies can change and influence the direction as well as the strength of the students' motivation, this section was thoroughly examined in the research.

Again, the examination started with the identification of statistically significant differences in the perception of the level of teaching in relation to the students' gender. The number of categories of the relevant variables implied the application of the Pearson's chi-squared test and its interpretation based on the corresponding *p*-value (at the significance level of α = 0.05). The results of this test are listed in Table 6.


**Table 6.** The perceived level of teaching at a university in relation to gender.

The test's result was calculated for three degrees of freedom, with the critical value being C = 7.815 (at the significance level of α = 0.05). In accordance with the corresponding *p*-value, the result is negative. This means that the dependence between the perceived level of teaching and the respondent's gender was not identified. Men and women perceived the level equally, which implies that during the designing and implementation of the measures for the increase of the perceived level of teaching, it is not necessary to take the students' gender into account.

The impact of the study program on the perceived level of teaching was analyzed as well. The results of the statistical testing, together with the relative frequencies, are listed in Table 7.

**Table 7.** The perceived level of teaching at a university in relation to the study program.


With three degrees of freedom and the critical value of C = 7.815, at the significance level of α = 0.05, the test did not confirm the presence of a statistically significant difference in the perceived level of teaching in relation to the study program. However, when looking at the relative frequencies, it can be seen that a slightly higher level was perceived by the students of management. To corroborate this tendency, it would be possible to broaden the research in the future, including a larger sample of students in it. An additional piece of information, shedding more light on the findings, is the fact that the students of management and informatics have several common subjects during the first year of their studies. Therefore, in future research, it would also be possible to filter out the impact of the common subjects on the perceived level of teaching. Subsequently, the finding obtained this way would be compared with the results of this research.

Finally, the influence of the year of study on the studied variable was analyzed. If the results from the previous section are followed, there is an assumption that while focusing on the impact of the study program and the year of study, the results obtained will be similar. The actually achieved results are listed in Table 8. The types of data entries and the number of categories led to the application of the Pearson's chi-squared test.

**Table 8.** The perceived level of teaching at a university in relation to the year of study.


Due to the same number of categories in the input data, the number of degrees of freedom as well as the critical value for the test did not change (three degrees of freedom, C = 7.815, significance level of α = 0.05). Based on the *p*-value reached, the test's result shows the presence of the statistically significant dependence between the perceived level of teaching and the year of study. This means that, in this case, the results differ when compared with those reached for the previous impact studied. The students of the first year perceive the level of teaching to be higher than the students of the second year. Within the overall concept of this research, these results represent a negative direction of the impact of the realized educational activities on the students' motivational readiness. From the perspective of the sustainability of the whole system, it would be desirable to focus on the causes of this effect more closely.

The aggregate results within the group of factors focused on the quality of the educational activities and their effect on the students' motivational readiness include these points:


#### *4.3. The Students' Expectations Related to Their Future Jobs*

The last group of factors affecting the students' motivational readiness is represented by their expectations and wishes related to their future careers. This group of factors is considered to be especially important since it is placed at the boundary between the education system and the labor market. In a similar way to the previous sections, the results obtained are at first presented aggregately, then they are structured into sub-sections created by focusing on the dependence between the given variable and the identification features of respondents. The aggregate results are captured in Figure 4.

**Figure 4.** The students' expectations related to their future jobs.

Based on the absolute frequency of the responses, the students most frequently expect their future employment to provide them with a friendly team of co-workers, delightful surroundings at the workplace, meaningful work and work tasks, and sufficient opportunities for their self-fulfillment. On the other hand, the least frequent expectations in relation to the future jobs were represented by the career advancement and by fringe benefits. The aggregate results reflect the opinions of the current students who, after finishing their studies, will enter the labor market. Therefore, the employers can use them as inspiration, for example, while creating and promoting new job offers.

Other findings are created by the analysis of the results from the perspective of detection of the dependence between the students' expectations and the identification features. The first identification feature was gender of respondents, the same as in the previous sections. Based on the type of the data and the number of individual categories, the z-score was calculated here. Its results, together with the relative frequencies, are listed in Table 9.

**Table 9.** The students' expectations related to their future jobs, with the dependence on gender being studied.


A statistically significant dependence was detected for the following expectations: Delightful surroundings, fair remuneration, and fringe benefits. In all three cases, the expectations are more frequent among women. In a similar way to the differences identified within the reasons for starting to study at a university, the causes of the situation here can also include the socially-conditioned different behavior of women and men. The possible implications of these findings are applicable on the side of employers. In the case when the employers want to attract more women to secure the diversity of working teams, they can focus on the identified factors in their job offers.

Another part is focused on the impact of the study program on the expectations related to future jobs. The nature of the study programs themselves implies certain estimated differences. Confirmation of these differences based on the z-score and the corresponding *p*-values (at the significance level of α = 0.05), together with the relative frequencies, are captured in Table 10.


**Table 10.** The students' expectations related to their future jobs, with the dependence on the study program being studied.

The test results confirm the estimated different expectations, specifically within the opportunity for self-fulfillment and career advancement, which are more often expected among the students of

management. These two factors are more important for future managers regarding the character of this work. An important finding is that independently from the study program, the opportunity for further learning and development is not often expected by the students. In relation to the current global environment, where the emphasis is being put on the life-long learning, this result again opens space for strengthening the motivational readiness of students as a consequence of suitably designed recommendations for the changes in the education process.

Finally, the dependence between the expectations related to future jobs and the year of study was analyzed. Since the data had the same type as the data entries in the previous case, the same procedure was applied again. Its results, accompanied by the *p*-values (at the significance level of α = 0.05) and the relative frequencies, are listed in Table 11.


**Table 11.** The students' expectations related to their future jobs, with the dependence on the current year of study.

The results show that, once again, the situation is not exactly the same as it was described for the impact of the factor represented by the study program. This supports the importance of studying the dependence of the results on the year of study separately. Based on the z-score calculated, the dependence between the expectation in the form of the delightful surroundings and the year of study was identified. Specifically, this expectation was more often among the students of the second year (88.89% of them) in comparison with the students of the first year (71.85% of them). This makes the particular expectation at the top of the list among students of the second year. This expectation can be considered to be one of the additional ones that the job seeker shall focus on only after satisfying his/her fundamental needs related to the work itself and the career path. This unexpected result, when the delightful surroundings are expected more often than e.g., the fringe benefits or the opportunities for the career advancement, elicits the need for deeper examination of the causes of this state in future research.

Overall, the results within the last group of factors affecting the motivational readiness of students revealed the following facts:


Based on these results, the research hypothesis was confirmed, following the specific differences described in the corresponding sections. These findings were taken into consideration while designing the recommendations for future improvement.

#### **5. Discussion**

The students' motivation to study at a university is affected by various expectations, motives and factors, including, for example, the opinions about the quality of education at a particular university, place of permanent residence [83], and others. This topic was also studied in the work done by Weberova et al. [84]. Based on the research presented in this article it was revealed that the most frequent motive for starting to study at the particular university is the effort for the increase in one's chances in the labor market. This is also corroborated by the statement of Chodasova et al. [85], saying that in the current time of globalization, education as a form of investment in human capital is an ideal platform for the improvement of the position of an unemployed person in the labor market, including its effect in the long run. Women in the survey chose this motive more often than men, which can be caused by the fact that, in the Slovak Republic, women are considered to be a disadvantaged group within the labor market, and they are trying to enhance their position via education.

On the one hand, the level of teaching is being affected by the students' approach to studying, but on the other hand, the key role is played by the teacher, including his/her competencies, willingness to continue learning new things [35,86], his/her approach to students, willingness to implement the latest knowledge into the subjects' content, and to use modern teaching methods. Teaching methods represent an important tool for the realization of the education process. The selection and suitable application of a method shall reflect the students' needs and it shall also react to the current society-wide trends in technical [87] as well as economic development. The selection of appropriate methods is also determined by various factors, such as the number of students in the study group, spatial and technical conditions, motivation of students for studying, professional level and experience of the teachers, and last but not least, by the quality and accessibility of the didactic tools and the supporting studying materials. Within this research, it was revealed that only slightly above 50% of students consider the level of education activities to be high (regardless of their gender, study program or the year of study). Therefore, there is still a huge potential for further improvement of the teacher's work and for the improvement of the content as well as the form of education and its particular activities.

It was revealed that among the students of informatics, regarding their expectations in relation to their future career, the most motivating factor is represented by a friendly team of co-workers, followed by delightful surroundings and meaningful work and work tasks. Among the students of management, meaningful work was substituted by the opportunities for self-fulfillment, which is probably a consequence of their ambition of getting managerial positions after finishing the studies. It seems that the issue of remuneration and fringe benefits is not that important for current students. This can be caused by the fact that the students of the first two years of the studies participated in the research, for whom the question of independent funding of their own needs is still a bit distant from the time perspective. Another reason can be the fact that despite the low average salary level in Slovakia, the graduates from universities, who find employment as IT professionals and managers, can achieve remuneration that is above average.

Within a wider context, the macroeconomic effects from the investment in individual components of human capital, for example, in the form of university education, lie in the increase and the sustainability of multi-factor productivity and macroeconomic performance, increase in incomes for the public budgets, growth of the life standard of citizens, and finally, in the growth of the knowledge level of people in the whole country [65,88]. In the case when the country does not pay sufficient attention to the education of citizens, economic inefficiency occurs [89], accompanied by the dissatisfaction of citizens and their possible emigration abroad.

Based on the research presented in this article, performed using the technique of a questionnaire which was created utilizing the inspiration from a particular study [65], it was revealed that young people are motivated to study at the particular faculty due to a better chance of finding employment and getting a higher salary and qualification. The motivation to study at the faculty due to better employment in the labor market was more often perceived among women. Despite the fact that the level of teaching is considered to be high by almost 50% of the students, regardless of their gender, study program or the year of study, the motivation to study is also supported by the students' expectations related to the jobs they will do in the future. The students of informatics expect mainly a friendly team of colleagues, delightful and stimulating working conditions, and the possibility of doing meaningful work. For the students of management, the third most often expectation is the opportunity for self-fulfillment. In relation to other factors, the differences based on the gender, study program, or the year of study were not statistically significant.

The greatest motivation for young people to study at the particular faculty is a better chance of getting employment in the labor market. For better employment of graduates in the practice, it is necessary for the students to have the opportunities to connect knowledge with practice during their studies. This is an interesting source of motivation for them to learn because it contributes to a more accurate idea about their future. In teaching, the teachers need to connect the latest results of science and research from the world and from the university with examples and forms of their application in the practice even more. The students are also more inclined to accept and absorb the information from a teacher who has experience also from the outside of the school environment, or who is working on the projects assigned from the business environment.

Based on the results obtained, several measures for the Faculty of Management Science and Informatics are designed within this research, which can serve as an inspiration for other faculties and universities, even beyond the borders of the Slovak Republic. The implementation of the measures recommended can increase the motivation of students to study at the university, and specifically at the particular faculty, which will secure the sustainability of education and strengthen the reputation of the faculty among the public audience. By employing highly motivated graduates, enterprises will acquire employees with a high level of human capital. This will also contribute to increasing business value generation for the stakeholders.

If students perceive that they can achieve an advantaged position in the labor market and that their possibility of getting favorable employment is strengthened, their motivation to start studying and to study successfully, even at the particular faculty, will be increased. This will create synergy among all three areas studied: Value generation in an enterprise (as an employer), the sustainability of education, and the students' motivation.

#### **6. Conclusions**

Education, as one of the forms of investment in human capital, improves the starting position and the negotiation power of individuals in the labor market, enables them to get a higher salary, or achieve higher performance at work. The effects from the employment of highly qualified people or from the investment in human capital realized at the level of enterprises lie, for example, in the increase in the production's quality, or in the increase of the productivity and performance. The investment in human capital and its value, and the ability to efficiently utilize the human capital available, also represents an important aspect of value creation and an enterprise's competitiveness in the current dynamic, open markets. The increased care of the employees via the investment leads to higher satisfaction with the work an employee does. For the employee, this represents higher motivation for performing his/her work tasks with higher quality, and it increases his/her loyalty to the enterprise. The enterprise gets qualified, healthy, and educated employees, which increases the value of human capital of the employees and the value for the stakeholder groups of the enterprise. This way the enterprise acquires an advantage against its competitors in the market space. All of this again contributes to the sustainability of education in the country.

When approaching this from a more specific perspective, attention can be paid to the motivation of students at universities. This is being influenced by various factors. The satisfaction of students and the sustainability of education depends on the fulfillment of these factors. Student's motivation can be affected by experience from the past, opinions of friends and acquaintances, reputation of the university, situation in the labor market, etc. The article was focused on the reasons leading the students to study at a university, the level of the teaching, and the students' expectations related to their future careers.

The recommendations for the faculty include paying heed to continuous updating of the content of the study programs being provided and to their consistent connection with the actual requirements of the labor market. These are the requirements of the enterprises as future employers of the graduates. These enterprises have a chance to get employees with high level of human capital, which will increase their value in the market. Since this factor was statistically higher among women, there is a chance to increase the motivation of girls for studying informatics at the faculty. The increase of the number of girls studying informatics currently belongs to the aims of other faculties too, and it is also a national as well as an international effort.

The level of teaching affects the resulting quality of education at universities. During the education process, the students directly interact with the teachers. The teacher is supposed to be competent and able to appropriately (e.g., via experiences and examples) explain a specific topic to a group of students. Teachers can apply various teaching styles in the process of education. Another factor of the education's quality is the portfolio of the methods utilized. When utilizing a suitable combination of different modern teaching methods accompanied by the right alignment of the learning styles of the students with the teaching styles of the teachers, the interest of the students in studying is encouraged, together with their creativity, and their critical thinking and expert argumentation are improved. This way, it is possible to enhance the quality and attractiveness of the university education as well as the readiness of the students not only for the successful passing of the exam within the given subject and the defense of the thesis, but also for the solving of common-life and work situations. This will create a precondition for the successful employment of students outside the school, thus in the practice. On the one hand, this will contribute to the sustainability of education at the faculty. On the other hand, successful graduates with high level of human capital will be a valuable asset for the enterprises in which they will be employed, and they will be able to considerably contribute to the generation of value for the enterprises' stakeholder groups. It is necessary to implement all of this within the conditions of the faculty and the university as well. It is helpful if the teachers themselves work on increasing their qualification, professional level and competencies, and they learn new and modern methods of teaching. It is important that the teacher is able to identify his/her preferred style of teaching, and after considering its advantages and disadvantages, he/she is able to use other styles as well, reacting to the learning styles of the students. It is also necessary to thoroughly and adequately often update the content of the subjects in the study program, in harmony with the world-wide trends in the specific professional fields.

Another recommendation focused on the increase in the quality of education at the faculty is to establish a program of regular training courses, focusing on new, progressive, and participative education methods, which shall be mandatory for all teachers. All of this will contribute to the sustainability of education at the faculty.

The motivational factors of the students also include the expectations related to their future jobs, being connected to a friendly working team, working conditions, meaningful work, and the opportunity for self-fulfillment. Therefore, the recommendation for the faculty's management and its employees is to regularly present these expectations at common work meetings, during specialized activities as well as at scientific conferences. These meetings include the gatherings of the IT cluster, in which the faculty represents a respected member, together with important employers and institutions of the labor market within the field of information technology. If personal managers can attract clever university graduates and create suitable working conditions aligned with their expectations, it will result in the increase in value generation in enterprises and in the increase in their competitiveness in the market.

Future research of the students' motivation can be focused on other factors and the research sample can be broadened by including students from higher grades. Since the situation in the labor market is constantly developing over time and the conditions in schools as well as in enterprises are changing, it is planned to repeat the survey regularly, after a certain period of time. Subsequently, in the future, it will be possible to perform the survey and comparison including students from partnered faculties or other universities in different countries as well.

**Author Contributions:** A.K. conceptualization, validation, project administration, writing the paper, writing—review and editing, visualization of the paper. M.M. data curation, formal analysis, visualization of the paper, writing the paper, writing—review and editing. E.M. methodology, data curation, formal analysis, visualization of the paper. M.D. investigation, resources, writing the paper. E.T. investigation, resources, writing the paper.

**Acknowledgments:** This work was supported by project APVV-16-0297 Updating of anthropometric database of Slovak population, Grant system of the University of Zilina, project VEGA 1/0382/19 Building a sustainable relationship with stakeholders of enterprise through value creation using ICT.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **A Cooperative Interdisciplinary Task Intervention with Undergraduate Nursing and Computer Engineering Students**

**Pilar Marqués-Sánchez <sup>1</sup> , Isaías García-Rodríguez 2,\*, José Alberto Benítez-Andrades <sup>3</sup> , Mari Carmen Portillo 4, Javier Pérez-Paniagua <sup>1</sup> and María Mercedes Reguera-García <sup>1</sup>**


Received: 16 September 2019; Accepted: 9 November 2019; Published: 11 November 2019

**Abstract:** This study proposed a collaborative methodology among university students in different grades in order to find sustainable strategies that are an added value for students, teachers, and society. In daily professional practice, different professionals must develop skills to collaborate and understand each other. For that reality to be sustainable, we believe that experiences must begin in the context of higher education. Social network analysis offers a new perspective on optimizing relationships between university students. The main goal of this study was to analyze students' behavior in their networks following an educational intervention and the association with academic performance, resilience and engagement. This was a descriptive quasi-experimental study with pre–post measures of a cooperative interdisciplinary intervention. Participants comprised 50 nursing and computer engineering students. We measured help, friendship, and negative network centrality, engagement, resilience, and academic performance. No significant differences were observed between pre–post-intervention centrality measures in the negative network. However, the help and friendship networks presented statistically significant differences between inDegreeN, OutDegreeN and EigenvectorN on the one hand, and resilience and engagement—but not academic performance—on the other. Academic performance was solely associated with the team to which participants belonged. Cooperative interdisciplinary learning increased the number of ties and levels of prestige and influence among classmates. Further research is required in order to determine the influence of engagement and resilience on academic performance and the role of negative networks in network formation in education. This study provides important information for proposals on sustainable assessments in the field of higher education.

**Keywords:** academic performance; cooperative learning; engagement; engineering; interdisciplinary learning; nursing; resilience; social network analysis; students

#### **1. Introduction**

The university context is essential for university students to be sensitive to behaviors aligned with sustainability [1]. Universities have considered joining the the Sustainable Development Goals (SDGs) as a strategic factor. They have carried this out through multiple organizations, such as the Sustainable Development Solutions Network (SDSN), the Environmental Association for Universities and Colleges (EAUC), the Association for the Advancement of Sustainability in Higher Education (AASHE), and the Australasia Campus towards Sustainability (ACTS). The SDGs are a set of priorities and aspirations to guide all countries to address the most pressing challenges in the world, including health and social welfare issues [2]. In this context and aligned with the definition of sustainable development [3], the concept of "sustainable assessment" emerges in the early years of the 21st century. Sustainable assessment is defined as an assessment "that meets the needs of the present and [also] prepares students to meet their own future learning needs" [4].

The main idea behind sustainable assessment is to prepare students to undertake assessment tasks that they will have to face during their lives [4]. Learning cannot be sustainable if it requires continuing information from teachers on student's work [5]. University students must build the capacity to become judges of their own learning; this includes self-assessment but also peer and collaborative assessment. In this sense, assessment must go beyond the idea of getting a mark for a given course: It must be seen as an educational tool and not as a simple measure or learning outcomes. In relation to sustainable assessment, the concept of "evaluative judgement" has gained attention today in higher education contexts, being defined as "the capability to make decisions about the quality of work of self and others" [6].

Learning in a collaborative environment is one of the scenarios where the ideas from sustainable assessment can be more beneficial, beyond the traditional ideas of summative and formative assessment [7]. Self-, peer-, and lecturer-based assessment can be used in conjunction in order to obtain a sustainable assessment system, as in the case of the authentic assessment for sustainable learning model [8].

Sustainable assessment is a field of current active research that needs to explore a plethora of possibilities; however, there are few applied studies on interventions that develop such sustainable practices [5]. We consider that studying networking in a collaborative and interdisciplinary learning experience is a good proposal within the sustainable assessment approach, as interdisciplinary collaboration during the future working lives of the students will be an everyday issue. To perform this kind of studies, teachers must design teaching–learning strategies that assess these collaborations. This research explores learning and assessment approaches based on interdisciplinary and collaborative work among students from the degrees of nursing and computer engineering, with an emphasis on studying the social relationships that are established during this collaboration and how these social interactions affect engagement and academic outcomes.

The period of adolescence is accompanied by changes in the socioaffective process that affect perception, ties between peers, and inclusion in social groups [9]. During this process, social influence between peers reaches its zenith, while the same influence exerted by parents begins to decline [10]. Peer relations in the classroom can promote the knowledge, skills, and social capital necessary to successfully transition from adolescence to adulthood [11]. In addition, the development of classroom relationships is intimately related to students' academic performance [12].

Understanding engagement and resilience in the classroom helps to elucidate the formation of ties between students. When forming working groups, the selection of team members by the students themselves improves engagement and motivation [13]. Furthermore, promoting engagement has positive effects on academic performance and reduces dropout [14]. Moreover, resilience helps students to solve problems [15] and enhances subjective wellbeing [16]. It is highly important to instil resilience in future health professionals [17]. Interpersonal behaviors can affect engagement and resilience as a result of the status generated and reputation processes within groups.

Classroom relationships generate a rich ecosystem of social ties that requires a theoretical research framework to gain understanding. Social network analysis (SNA) comprises a method for analyzing the structure of ties within a network, which is its main difference from other methods of analysis [18]. SNA is based on the idea that the ties between network participants are meaningful. Hence, these ties are analyzed to elucidate their significance [19]. A social network is a set of nodes, some of which are

linked by lines. The nodes represent individuals or groups, and the lines indicate that the nodes are connected among them, generating a social structure [20]. This network of relationships or networks transfers resources inherent to the structure generated among individuals [21]. SNA studies the contact that exists not only between the actors but also between their goals and objectives, since their objectives are achieved through connections and relational behaviors [18,22,23].

Friendship and help networks have been among the most frequently analyzed networks in SNA, while more recently, the negative network has also emerged as a subject of study. Analyses of the friendship network examine friendship ties between nodes and the degree centrality of their intensity [24]. Centrality is defined as the position of the actor in the network [25]. Findings are useful to determine the influence of friendships and ties between peers on the acquisition of new values and behavior modification [14] or the importance of the most popular students and their impact on the dynamics of relationships [26]. Analyses of the help network assess the connection and intensity between nodes when problem-solving or seeking advice [27]. Previous studies have confirmed the utility of the help network when seeking prenatal information in the absence of formal resources [28] or receiving support and acceptance from students in the case of speech difficulties [29]. In relation to negative networks, it has been demonstrated that nodes avoid interacting with other nodes [30], and researchers have underlined the influence of negative ties in the workplace [31,32]. Negative relationships refer to the intensity of disgust established between two nodes and to whether the person knows that the other person dislikes him or her [31].

The structural analysis of networks can be applied to various fields of study, such as business relationships [33], tourist travel intentions [34], resilience in disasters [35], and mental health [36]. In the field of education, the context of the present study, SNA has been applied to explore (i) the social influence of ties on adolescents' mental health [37], (ii) the social dynamics of groups in educational camps [38], and (iii) the formation of friendships between students from different ethnic groups [39].

Among engineering students, SNA has been used to determine the influence of networks on performance, demonstrating that a higher number of ties is associated with better academic outcomes [40]. In nursing students, SNA has been used to explore the influence of networks on engagement and resilience [41] and determine their role in the development of technological competence [42]. It has been found that the network perpective is suitable for the analysis of resilience, since it is developed in sociological systems [43]. The network perspective is focused on the study of the structure of these sociological systems, and the ability of that structure to be resilient. A command of information and communication technologies is not among the competences instilled in nursing. Gamification is considered a novel and interesting approach to the development of competencies in new technologies [44] and its use to instil computing skills in the nursing profession has proven effective as a means to achieve a better command of these technologies [45].

These issues motivated the present study of how the dynamics of social interaction are associated with classroom behavior when sharing ideas in order to achieve a good academic outcome. The interdisciplinary intervention aimed at determining nursing and computer engineering students' behavior that has not previously been analyzed and remains unexplored in the literature. Notably, our SNA included an analysis of negative networks between students, which has not previously been examined in this population. In particular, the recent literature contains few studies that have focused on students and the impact of engagement and resilience [41,46], and none of them included academic performance among their variables. The present study is the first to include SNA in an interdisciplinary intervention with university students.

The study objectives were:



The objectives allowed us to propose a cooperative and multidisciplinary framework, which will be sustainable for Nursing and Computer Engineering degrees.

#### **2. Materials and Methods**

This was a quasi-experimental descriptive study with pre–post-intervention measures.

#### *2.1. Sample Description*

Participants were recruited using convenience sampling [47]. Students from two undergraduate courses were approached and presented the project. One of the courses belongs to the fourth year of the Computer Engineering degree and the other one to the third year of the Nursing degree. These degree courses were taught on different campuses located 113 kilometers apart. As shown in Table 1, 26 students from the Nursing degree course and 24 students from the Computer Engineering degree agreed to participate in the project after being informed. Interdisciplinary work groups were formed with randomly selected students, so that there would be a similar number of nursing and engineering students in each work team.

**Table 1.** Descriptive data and comparison of pre–post-intervention centrality variables (N = 50).


The sample consisted of 50 students taking two different degree courses at a public university in Spain. All the individuals participated voluntarily in the study after being informed.

#### *2.2. Variables*

The variables analyzed were as follows (Table 2):



#### *2.3. Instruments Used to Collect Data*

Data on variables were collected by means of an online questionnaire viewable on any device (desktop, laptop, mobile device or tablet). The questionnaire was accessed via a URL, entering a username and password, and incorporated an automatic anonymization system. The server ensured secure data transfer via SSL encryption and HTTPS (Hypertext Transfer Protocol Secure).

The questionnaire included the following:


#### *2.4. Procedure*

Data were collected on two occasions: first in the initial face-to-face session and again on the day when the completed task was presented.

Descriptive variables, engagement, resilience, and marks were processed using Microsoft Excel. Structural variables of sociocentric network centrality were analyzed using square matrices for each network. It was necessary to dichotomize the data using intermediate encoding of the friendship, help, and negative networks. We used normalized data for centrality values in accordance with UCINET v. 6.666 [53] (Table 3).


**Table 3.** Dichotomization of network interactions.

#### *2.5. Intervention*

The interdisciplinary intervention consisted of dividing the class into nine teams of five or six students each from both degree courses; these teams were required to carry out a cooperative task on applications in the field of health. Participants were taught at different campuses located 113 kilometers apart, so they only met face-to-face at the initial session when the teachers presented the cooperative task. Their subsequent contacts took place via online networks.

The cooperative task involved three stages: (a) an initial face-to-face session to explain the task objectives and method and establish personal contact during a 5-hour session in a non-academic environment (a cafeteria with brunch included); (b) implementation of the task over the course of 40 days subsequent to the initial session, communicating by means of mobile phones, emails, and instant messaging; (c) presentation of the completed work by all members of the team via videoconference between the two campuses.

To complete the task, the nursing students had to explain a healthcare need, and the engineering students had to formulate a technological solution for the identified healthcare need. In their oral presentations, all team members were required to present part of the completed work but with the caveat that the nursing students had to present the technological solution and the engineering students had to talk about the healthcare need.

Assessment was achieved by both teacher and peer evaluation of the collaborative work. The nursing degree students evaluated the computing engineering students' presentations (except those from their own working group) and vice-versa: The computer engineering students evaluated the nursing students' talks. Academic performance was measured as the individual mark awarded for the results of the cooperative task (evaluated by the teacher), a multiple-choice test (for the peer evaluation previously described), and classroom participation.

#### *2.6. Ethical Considerations*

Participants were informed of the study objectives and method. Participation was voluntary, and students could cease to participate at any time. All personal data were processed ensuring confidentiality and anonymity. Simulated names were generated with the aid of the tool described in Benítez et al. (2017) and used to create the network graphs [54]. This study was approved by the University of León Ethics Committee (Ref. ETICA-ULE-026-2018) and adhered to the Declaration of Helsinki, Law 15/1999 of 13 December, on Personal Data Protection, and Law 14/2007, of 3 July, on Biomedical Research.

#### *2.7. Data Analysis*

We used the Kolmogorov–Smirnov test with Lilliefors correction to determine the normality of values for all the variables analyzed except those for centrality measures. The descriptive statistics are given as means and standard deviations.

To determine differences between pre- and post-intervention variables, we used the Student's *t*-test when distribution was normal and Wilcoxon's *t*- test when it was non-normal.

To determine correlations between parametric values, we used Pearson's correlation coefficient, while for nonparametric values, we used Spearman's correlation coefficient.

Significance was set at *p* < 0.05 and *p* < 0.01. All statistical analyses were performed using SPSS v. 25.0.

#### *2.8. Results*

Universities are the institutions of higher education (HEI) whose objective is to contribute to the sustainable transformation of societies through the training of future professionals. In this sense, this sustainability and transformation project must start from the lectures at the university campus, with specific actions in the curricula and oriented towards what UNESCO calls the "Focus of the whole school". The study of collaboration networks among students of different grades constitutes a strategy for fostering the future sustainability of professionals. In our case, it was proposed that a nurse and/or an engineer could improve patient care. This is based on the fact that a nurse knows what the patient demands, and the engineer knows how to propose a technological solution and integrate the use of technologies in health systems. To achieve this, the nurse and the engineer must develop effective communication channels, have critical thinking, and know how to work in networking teams with professionals from different disciplines.

If this method of working achieves positive outcomes among the students, it will possibly be achieved also in the future, when the students become professionals. Our intervention was related to sustainability because trying to demonstrate that interdisciplinary work solves problems responds to societal demands and achieves objectives effectively.

The application of SNA helped us to deepen the structures of collaborative networks among students. The specificity of the method allowed the identification of relational roles, such as Indegree, Outdegree, Eigenvector, and Betweenness. In addition, this research related these relational roles with fundamental constructs for teamwork, such as resilience and engagement, as detailed below.

Table 4 gives the descriptive data and a comparison of normalized network centrality variables pre- and post-intervention. We found that the help and friendship networks presented variables with significant differences in InDegreeN, OutDegreeN and EigenvectorN. By contrast, no significant differences were detected for any negative network variable.


**Table 4.** Descriptive data and comparison of pre–post-intervention centrality variables (N = 50).

Legend: InDegreeN, normalized InDegree; OutDegreeN, normalized OutDegree; BetweennessN, normalized Betweenness; EigenvectorN, normalized Eigenvector. \* Correlation is significant at 0.05. \*\* Correlation is significant at 0.01.

Graphic representations of pre- and post-intervention networks are shown in Figures 1 and 2 (help networks), Figures 3 and 4 (friendship networks), and Figures 5 and 6 (negative networks). Blue indicates men and pink indicates women, while squares denote nursing students and circles denote computer engineering students. Tie density in the help and friendship networks showed clear changes over the course of the study. Pre-intervention, most ties in these networks occurred between students on the same degree course, with hardly any ties between degrees. In addition, more interactions were observed between nursing than between computer engineering students, with the latter presenting greater selectivity when forming friendships or seeking help. Post-intervention, the number of interactions increased in the help and friendship networks, but differences still remained. Meanwhile, the pre- and post-intervention negative networks presented a series of central nodes that connected the majority of ties, and it was more difficult to distinguish between the two degrees.

**Figure 1.** Graph of the pre-intervention friendship network (simulated names).

**Figure 2.** Graph of the post-intervention friendship network (simulated names).

**Figure 3.** Graph of the pre-intervention help network (simulated names).

**Figure 4.** Graph of the post-intervention help network (simulated names).

*Sustainability* **2019**, *11*, 6325

**Figure 5.** Graph of the pre-intervention negative network (simulated names).

**Figure 6.** Graph of the post-intervention negativa network (simulated names).

Tables 5 and 6 show the behavioral change in engagement (subscales dedication, vigor, and absorption) and resilience, which was statistically significant. All the variables showed an increase following the educational intervention.


**Table 5.** Descriptive data and comparison of pre–post-intervention centrality variables (N = 50).

Legend: InDegreeN, normalized InDegree; OutDegreeN, normalized OutDegree; BetweennessN, normalized Betweenness; EigenvectorN, normalized Eigenvector. \* Correlation is significant at 0.05. \*\* Correlation is significant at 0.01.

**Table 6.** Comparison of pre–post-intervention engagement and resilence variables (N = 50).


\* Correlation is significant at 0.05. \*\* Correlation is significant at 0.01.

As regards correlations between the variables of centrality and engagement, we found relationships in the help and friendship networks, but not in the negative network. The help network was statistically significantly related to the three engagement subscales and, in particular, the relationship between BetweennessN and all three engagement subscales presented a level of significance of 0.001. The same behavior was observed between EigenvectorN and the absorption subscale of engagement (Table 7).


**Table 7.** Correlations between network cantrality variables and engagement (N = 50).

Legend: InDegreeN, normalized InDegree; OutDegreeN, normalized OutDegree; BetweennessN, normalized Betweenness; EigenvectorN, normalized Eigenvector; P, Pearson's correlation coefficient; S (Spearman's correlation coefficient). \* Correlation is significant at 0.05. \*\* Correlation is significant at 0.01.

Relationships between resilience and the centrality variables of all networks were statistically significant for half of the variables in the help and friendship networks. Both networks presented an association with the normalized OutDegree and normalized Betweenness. In particular, the relationship between the help network and normalized Betweenness presented a significance of 0.001. Once again, no statistically significant relationships were observed for the negative network (Table 8).


**Table 8.** Correlations between network centrality variables and resilience (N = 50).

Legend: InDegreeN, normalized InDegree; OutDegreeN, normalized OutDegree; BetweennessN, normalized Betweenness; EigenvectorN, normalized Eigenvector; P, Pearson's correlation coefficient. \* Correlation is significant at 0.05. \*\* Correlation is significant at 0.01.

The only statistically significant relationship observed between the sociodemographic variables (sex, degree, team) and centrality variables for the three networks and academic performance was identified between the team and academic performance (r = 0.283; sig. 0.046).

#### **3. Discussion**

In the present study, we conducted an educational intervention based on a cooperative interdisciplinary task carried out by undergraduate nursing and computer engineering students. The intervention was implemented on a management course (nursing) and a semantic modeling course (computer engineering). It was based on mixed working teams where the nursing students had to identify and explain a health demand and the computer engineering students had to formulate a technological solution. Assessment of the task formed part of each participant's final mark for academic performance.

We determined changes in the students' help, friendship, and negative networks through measures of centrality, engagement and resilience. These assessments were performed pre- and post-intervention. As regards the analysis, we correlated centrality, engagement, resilience and sociodemographic variables with the final mark for academic performance in the courses analyzed. Our main finding was that the centrality variables in the negative network did not present any statistically significant changes after the cooperative task intervention. However, we did observe statistically significant results for engagement, resilience and centrality variables in the help and friendship networks, except for Betweenness centrality. We obtained correlations between help and friendship network centrality variables and engagement and resilience. With respect to academic performance, we only found a correlation for the variable "team".

In relation to the educational intervention, we found that the cooperative task generated changes in student behavior. These findings are consistent with those reported in similar studies using team work to help students to acquire subject competencies [55], especially in health education teaching placements [56] and crossdisciplinary workshops in universities about different sustainability issues [1,2]. As regards the study population, we did not find any studies that analyzed a population as disparate as our sample (nurses and computer engineers), although some analyzed distinct but similar populations such as medical and nursing students, using a model of interprofessional problem-based learning to assess the effect on learning (mutual understanding of roles, appreciation, and interprofessional communication and collaboration) [46,57]. In this respect, our study incorporated a novel and risky interdisciplinary perspective, but with considerable success. The technological complexity of health care is rising, and this will mean that professionals in various areas of knowledge must know how to work in teams.

The main contribution of our study is the application of SNA as a quantitative methodology to determine relational behavior change. Our intervention required students to interact in order to carry out an academic task for subsequent assessment. Hence, we generated a need on the basis of which each group of students had to establish relations of interdependence to achieve a shared goal: to complete the academic task and obtain a good academic outcome. The literature indicates that relationships emerge when there is a shared goal or vision, but the results suggest that SNA applied to collaborative learning has not reflected this same diversity of actors and relational ties but instead

has solely explored one-mode networks of learners connected by communication-based relational ties and has been limited to a descriptive report of SNA results [40,58]. However, SNA facilitates a structural interpretation of relationships in undergraduate learning and their impact on learning outcomes, which can inform educators in unique ways and improve educational reform [59].

We found no evidence in the literature review of any study that had conducted this type of educational intervention with the variables analyzed, and it is therefore not possible to perform a direct comparison. SNA has been used as an instrument to analyze or explain interdisciplinary behavior in the field of education [33,60]. Our research yielded associations between students and change in friendship and help but not negative networks. Both the friendship and help networks presented significant differences in the centrality variables of InDegree, OutDegree, and Eigenvector, suggesting that cooperative work with a shared goal exerts the most influence on the number of ties received and emitted and the capacity for influence in the classroom. By contrast, the negative attitude of avoiding a peer, measured by means of the negative network (which of the following classmates do you avoid interacting with? [30]), did not lead to changes following our educational intervention. One explanation for this may be that a negative perception of someone is not a barrier to working together in a team when there is a shared goal which requires interaction. Put more simply, you might not like working with someone, but when you have to in order to achieve a goal, you are capable of working efficiently with the people around you. These findings are in line with those reported by Wang et al. (2015), who proposed a machine learning algorithm for predicting positive and negative relationships in social networks. They found that both types of relationships, which include support and opposition, and trust and suspicion, are present in all networks, and that research in this context could help us to understand the formation of relationships and network structures. In turn, this would enable us to optimize them in order to achieve shared goals.

As regards the significant change in the help and friendship networks, this may be associated with two factors. First, the cooperative task intervention enhanced network contacts, because members shared experiences, frustrations, and achievements, furthering cohesion. In this respect, our findings are consistent with those of studies by Dreier-Wolfgramm et al. (2018), who showed that their problem-based learning intervention with nursing and medical students had increased each participant's level of knowledge through contact with the others and had exerted a positive effect in terms of mutual appreciation [57]. Second, cooperative work enhanced the students' self-esteem because it helped them to clarify doubts, solve problems, and achieve their goals. In other words, the intervention generated friendly sociability. An earlier study with adolescent students found that self-esteem heightened sociability, reduced symptoms of depression and sadness, and increased the number of ties with other students [37].

The graphical representations of the networks show that the number of contacts increased over the course of the intervention and that the students mainly formed groups within their discipline rather than according to sex. Thus, nursing students interacted more with each other, as did computer engineering students. This could be explained by homophily, the tendency of individuals to interact with others similar to themselves [46]. In this context, similarity would refer to the same team, race, sex or degree course. In our study, nursing students shared a common context and language in terms of health, while computer engineering students shared a context more related to technology. Previous studies of networks have found that the main variable associated with the closest ties in a class of nursing students was sex [41]. However, in this study, it was not possible to extrapolate the data in the graphics as regards sex because our sample was not homogeneous in terms of the number of men and women on each of the degree courses.

Our findings indicate that cooperative work significantly changed the psychological variables analyzed, which is consistent with the results of previous studies. With regards to university student engagement, authors such as Persky (2012) observed this change in team-based learning in a foundational pharmacokinetics course [61]. Similarly, Promo et al. (2018) suggested that incorporating tasks based on interdependence can promote engagement in small teams as well as in an entire class of

undergraduate students [62]. In relation to resilience, our study highlights its role in helping university students to overcome adversity and learn from experience. Thomas and Asselin (2018) advocated strengthening resilience in order to improve clinical placements and promote support, education, and reflection in the context of university clinical education [63].

The post-intervention results obtained for the three subscales of engagement were 3.4 ± 1.3 for dedication, 3.4 ± 1.3 for vigor, and 4.2 ± 1.3 for absorption, which differ slightly from those reported in similar studies. One study conducted with 90 nursing degree students obtained means of 4.4 for dedication, 3.08 for vigor, and 3.21 for absorption [46]. Another study conducted with 134 nursing degree students obtained means of 4.82 for dedication, 3.13 for vigor, and 2.98 for absorption [41]. In both cases, it can be seen that the subscale of dedication obtained the highest means. However, in our study, the subscale of absorption (an individual's capacity to be totally focused on work) obtained the highest mean. One explanation for this finding might be that since our students were studying two different degree courses taught on different campuses, they had to overcome the obstacle of geographical distance, optimizing the time invested in work without distractions.

The result obtained for resilience post-intervention was 29.82, similarly to other studies on nursing students, with results between 28.6 and 34.7 [41,46,64]. Achieving a high degree of resilience is important because it is associated with experiencing less psychological distress and, above all, less academic burnout [65]. This may be explained by the high levels of stress experienced during university studies, which could diminish undergraduate nursing and computer engineering students' preparedness to exercise their profession. Previous studies of nursing students have found that burnout during nursing education predicts lower occupational preparedness and future clinical performance, together with high stress levels in new students [65,66].

As regards associations between the variables of centrality and engagement, we observed numerous statistically significant correlations in the help and friendship networks, with the exception of Betweenness. Similarly, Fernández-Martínez et al. (2017) observed the same behavior in 48 first-year nursing students [41]. Relationships between the variables of centrality in the friendship and help networks were associated with resilience, in contrast to the findings of previous studies with nursing students, in which resilience was only related to the friendship network [41]. In addition, it should be noted that we found relationships between the negative network and engagement or resilience, suggesting that negative ties involving enduring and recurrent negative judgements and feelings do not prevent actors from performing a task. Other researchers have claimed that workplace ties are "friendly", "positive", or at least "neutral", and that although occasional upsets may arise, creating temporary discontent with individual or team achievements, positive ties transcend negative ones, canceling the latter's effects on the actors [32].

Lastly, academic performance did not correlate with any variable except the team to which each participant belonged. One explanation for this result is that the teams obtained different marks and, therefore, students on the same team would have similar marks because the team mark contributed to individual marks. This was a surprising finding because previous studies on nursing students have reported that a better result for the three subscales of engagement was associated with better academic performance [67]. Another study found a similar result for adolescents, concluding that promoting engagement in adolescence would lead to better performance in high school [14]. The means obtained for the engagement subscales might have been affected by student motivation. Since our students were studying different degree courses, they did not know each other and, consequently, could not select their own teams, which were assigned by the researchers. This may have influenced engagement, although in a positive sense via the formation of new networks. In turn, students with greater OutDegree centrality tended to present a higher level of engagement [13]. Social prestige as measured by students' Eigenvector has been associated with better academic outcomes in some—but not all—studies [12,68]. In our study, performance in the group task did not present any statistically significant relationship [69]. The importance of the results obtained is not only in the replicability of the interventions in different educational contexts, but also in their sustainability over time, to be

able to respond to the problems and demands generated by the students. These conflicting results in the literature indicate the potential importance of social prestige arising from the formation of social networks. In the recent literature, as a measure of social capital, the Eigenvector has been positively associated with academic outcome, as has InDegree [70]. Nevertheless, none of the networks obtained in our study presented any correlation with academic performance, possibly because we used the final mark as our measure of performance, which included other criteria in addition to the cooperative task, such as multiple-choice test results and classroom participation.

Our study presents several limitations. First, we did not include more variables that might have better elucidated the results obtained. Secondly, we did not include a control group undergoing a similar experience in a different form, and it was therefore not possible to compare differences between a control and experimental group. However, the inclusion of all the students and the creation of similar teams generated added value when interpreting the data. Another limitation was the correlation only between performance and equipment. This should be considered for future research so that the yield variable includes evaluations of relational competencies. This study did not aim to generalize or prove cause–effect relationships but to test some innovative and initial hypotheses that will be further tested at a larger scale in a future study with other degrees.

#### **4. Conclusions**

One of the most important premises of the context of high education is that the focus of the teaching–learning processes is oriented to the demands of society and developed in a sustainable way. In our case of health sciences and computer engineering, society increasingly demands that we complement and understand each other. That is, the nursing professional captures the demands of the patients and must be competent to evaluate the cost–benefit of the proposals to meet their needs. On the other hand, the evidence demonstrates that technological applications can be useful for patient care and save costs, but nurses are not trained to make those proposals. Computer engineers do have the core of knowledge to provide technological solutions.

This way of understanding, through collaborative work between different faculties, how the university context approaches social reality, is a useful way to propose innovative and sustainable solutions in teaching–learning.

In the intervention carried out in this research, students showed that they were able to work in a network, although their areas of knowledge were very different—nursing and computer engineering.

The behavior of nursing and computer engineering students following a cooperative task intervention changed significantly for all network centrality variables in the help and friendship, but not negative, networks.

In addition, we graphically represented these network changes, showing that ties were basically formed between students taking the same degree course.

We observed a significant change in engagement and resilience following the cooperative interdisciplinary task intervention.

We analyzed centrality, engagement, resilience, and sociodemographic variables and academic performance and found that many of the centrality variables in the help and friendship networks were associated with engagement and resilience. However, academic performance did not correlate with any variable analyzed except the team to which the students belonged.

As a practical application, teachers should identify engagement, resilience, and networks in order to improve the communication skills necessary to carry out cooperative tasks, an essential aspect of the students' future work.

Currently, the world is conceptualized as global. In this sense, we must also conceptualize the work between professionals from different fields, different organizations, and different countries as collaborative and global, given that the pace of society demands it, and technologies allow it.

We believe our analysis of an interdisciplinary teaching intervention provides some valuable evidence that can suggest future strategies for use by university teachers to develop, in students, important skills needed by professionals, in the context of a sustainable lifelong learning framework.

**Author Contributions:** Conceptualization, P.M.-S., I.G.-R., J.A.B.-A., J.P.-P., and M.M.R.-G.; data curation methodology, J.P.-P., M.M.R.-G., J.A.B.-A., P.M.-S., M.C.P., and I.G.-R.; software, J.A.B.-A., M.M.R.-G., P.M.-S., I.G.-R., and M.C.P.; validation, P.M.-S., I.G.-R., J.A.B.-A., M.C.P., and M.M.R.-G.; formal analysis, M.M.R.-G., P.M.-S., I.G.-R., J.A.B.-A., M.C.P., and J.P.-P.; investigation, P.M.-S., I.G.-R., J.A.B.-A., M.C.P., J.P.-P., and M.M.R.-G.; resources, P.M.-S., I.G.-R., M.M.R.-G.; J.A.B.-A., M.C.P., and J.P.-P.; data curation, J.P.-P., M.M.R.-G., P.M.-S., I.G.-R., J.A.B.-A., and M.C.P.; writing—original draft preparation, M.M.R.-G., J.P.-P., P.M.-S., I.G.-R., J.A.B.-A., and M.C.P.; writing—review and editing, J.P.-P., P.M.-S., I.G.-R., J.A.B.-A., M.C.P., and M.M.R.-G.; visualization, P.M.-S., I.G.-R., J.A.B.-A., M.C.P., J.P.-P., and M.M.R.-G.; supervision, P.M.-S., I.G.-R., M.M.R.-G., J.A.B.-A., M.C.P., and J.P.-P.

**Funding:** This research has had partial funding from the Master of Socio-Health Research (University of Leon). **Acknowledgments:** To the students and professors of computer science and nursing of the University of León. **Conflicts of Interest:** The authors declare no conflict of interest.

## **References**


© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Review* **A Matter of Responsible Management from Higher Education Institutions**

#### **Nicolas Roos**

Faculty of Business Management and Economics, Technische Universitaet Dresden, 01069 Dresden, Germany; nicolas.roos@tu-dresden.de

Received: 1 October 2019; Accepted: 13 November 2019; Published: 18 November 2019

**Abstract:** Higher education institutions (HEIs) are influential social institutions which disseminate knowledge, promote innovation, and educate future decision-makers. The increasing awareness of HEIs as social actors has increased the pressure on them to accept and act upon their social responsibility. Processing this responsibility requires a structured management approach. The little attention given thus far to management performance and structured steering processes of social responsibility in HEIs marks the research gap the present study is focused on. This article provides a systematic review of scientific and academic publications, applying the concept of Social Performance after Wood (1991). The study aims to combine different research and modeling approaches to examine individual elements of social performance along the dimensions of processes of social responsiveness and outcomes of institutional behavior. With this approach, the study aims to answer the question of how HEIs assume their responsibilities as social institutions. The results show that observable outcomes of social behavior in the academic environment reflect a broad understanding of different approaches. By clustering the encoded literature into processes and outcomes, the study structures the fragmented body of research reflecting the various characteristics of the higher education sector.

**Keywords:** responsible management; higher education institutions; social performance; institutional theory; systematic review

#### **1. Introduction**

Higher Education Institutions (HEIs), as influential social institutions, act as promoters of change through research and education [1]. Within this function, they create and disseminate knowledge, emphasizing their role in social capacity in society [2].

With a general dissemination of business tools in public institutions, the proliferation of responsible management practices likewise increases the awareness of effects of organizational activities within HEIs [3]. The orientation towards a higher level of awareness through improvement based mechanisms and performance enhancement leads to a professionalization of HEIs' management structures [4,5]. This constitutes an essential capacity for providing resources to process responsibility [6].

Since structured management appears to be a critical success factor for operating responsibility, the question of systematically approaching management performance and steering social sustainability activities defines a research gap currently not being addressed in the scientific literature.

Previous studies approach the implementation of responsibility [4,7] or the integration of stakeholder interests in the context of sustainability considerations [8,9]. Other authors address responsibility by examining frameworks and indicators [10–13].

Though measuring effects and outcomes is an important aspect, this forgoes the idea that controlling and managing activities require a superstructure to steer these practices. So far, operations for processing responsibility have received little attention in the literature [14–17].

This paper seeks to determine the efforts of HEIs practicing social sustainability and responsible management. For this case, the study applies a systematic literature review based on the model of social performance after Wood [18].

The review pursues the research question of how HEIs assume their responsibilities as social institutions within the dimensions of social performance of Wood (1991). The objective of this study is to apply a structured approach to HEIs' management activities concerning social sustainability, which consequently makes it feasible to apply tools for steering. Accordingly, the study makes two contributions to the literature stream of responsible management and social performance at HEIs. First, archival data research provides a welcomed opportunity to analyze current research approaches and systemize the status quo of how HEIs manage their social responsibility. In this regard, the review is useful for mapping current management performance in that research field and identifying further research areas not yet studied [19]. The review adds to the mostly qualitative and case-based research in this area, reporting on implementation approaches [19–21] or discussing factors of success [17,22,23], by proposing a pattern for the assessment and management of social performance issues in HEIs.

Second, the study expands the existing literature on sustainability in HEIs by investigating the managerial perspective on practicing social responsibility. The study highlights the role and function of social performance management after Wood [18] and applies this to the context of HEIs. This enables new perspectives on aspects of systematic steering and processing organizational responsible behavior, which contributes to a better understanding of balancing local and global demands, and to closing the gap between social values and organizational behavior in order to improve the social legitimacy of HEIs.

The remainder of the paper is structured as follows: the study continues with a more detailed theoretical background on HEIs, responsible management, and social performance after Wood in Section 2. The methodology, the research design, and measurement of constructs are explained in Section 3. In Section 4, the results of the study are presented. Finally, the study concludes with findings, and presents limitations and implications for future research, in Section 5.

#### **2. Theoretical Background**

In the last 20 years, social responsibility has been considered relevant for nearly all entities providing services to the public, including HEIs [20]. HEIs, as public institutions with a primary mission revolving around teaching and research, have an emphasized role in society, since they develop professional skills and knowledge for conscious acting and foster research on sustainable solutions and innovations. This social mission, being embedded within the responsibility to act in the general interest of society, produces social legitimacy and forms the HEI's license to operate. Creating legitimacy by fulfilling a social mission and responsible acting more or less describes a management task, with assessment being a vital component that enables its success. Anticipating this, an increasing number of coordinating bodies have emerged in HEIs [11]. Although the assessment of research and teaching performance is a field that has received significant attention, the area of managing social responsibility in the course of fulfilling social legitimacy has not [17]. The relatively low proliferation of action being taken is grounded in a diverse understanding of the term social responsibility, which goes back to the lack of a generally accepted and common definition [24–27]. Vasilescu et al. [28] approach the social responsibility of a university as "the need to strengthen civic commitment and active citizenship; it is about volunteering, about an ethical approach, developing a sense of civil citizenship by encouraging the students, the academic staff to provide social services to their local community or to promote ecological, environmental commitment for local and global sustainable development". Others define the social responsibility of HEIs from a stakeholder-oriented perspective [22,29,30]. In another approach, Reiser [31] identifies distinct types of social responsibilities: (1) organizational responsibility to lead as an ethical example, (2) cognitive responsibilities of scientific output, (3) social responsibilities towards the community on issues of social development, and (4) educational responsibilities for responsible citizens.

Since this categorization delivers a suitable approach for understanding the different types of responsibilities of institutions, the examination applies this as a working definition for the examination of responsible management from a managerial perspective, although the study also investigates the implementation of type (1) and (3) responsibilities. The study furthermore follows the assumption that defining social responsibility should deliver information on the purpose of acting, how to pursue this purpose, and the ethical obligations serving as guiding principles [32].

To avoid confusion with environmental topics, the study excludes them from the present study in order to produce a clear construct. Consequently, the study applies the term responsibility instead of sustainability [32]. Following this working definition, the study aims to seek a proper tool for the identification of certain activities. With a focus on management activities and steering mechanisms, the application of performance-driven frames focusing on social issues provides appropriate tools for the subsequent analysis.

The application of Wood's model [18] on social performance (see Table 1) enables a mapping of existing management efforts around social issues. The model serves as a framework to reveal social performance in HEIs' management, since it delivers an open structure to map the scattered field of activities in HEIs. Thereby, Wood's model is well-suited for systematically reviewing the literature along internal and external perspectives, as well as processes and outcomes. This enables a wide focus, in order to depict a broader spectrum of activities, with implications for the non-observable issues of organizations (as proposed by principles of CSR). On the other hand, the systematic review of the literature serves to check the applicability and practicability of Wood's model for theory and practice, since it demonstrates whether responsible management performance follows an underlying structure (as proposed by the model), which makes it assessable for management bodies.

In order to discuss these questions, the following section describes Wood's model of social performance in detail. Wood's model goes back to Carroll's approach [33] to mapping the social performance of organizations. In his model, Carroll describes three integral parts of social performance, which are split into the following dimensions:

(1) Corporate Social Responsibility (CSR) as legal, ethical, or economic responsibilities;


Complementing his thoughts, Wartick and Cochran [34] modified Carroll's model by restructuring the dimensions of social performance based on the assumption of an interaction between the principles of social responsibility, the process of social responsiveness, and the policies addressing social issues [34]. One major contribution of this modification is the recognition that social issues need to be actively managed. In response, Wood developed the model of social performance as an " ... organization's configuration of principles of social responsibility, processes of social responsiveness, and policies, programs, and observable outcomes as they relate to ... societal relationships" [18].


**Table 1.** Corporate social performance according to Wood (1991) (own illustration).

According to this understanding, Wood defines the dimensions of social performance as follows: (1) Principles of corporate social responsibility describe basic motivations and expectations on different levels of consideration. The principle of legitimacy justifies responsible acting on an institutional level, as there are general expectations from society for all kinds of entities.

Public responsibility describes the expectations towards distinct forms of organizations, which is exercised by managerial discretion on the individual level. At this level, responsibility is understood as a moral task, which cannot be completely determined by the organization. According to Wood, the principles are not observable features of an organization.

(2) The processes of corporate social responsiveness describe action patterns contrasting normative principles of CSR. According to Wood, processes are indirectly measurable by inference [18]. Furthermore, Wood proposes the application of a three-stage management process, consisting of environmental assessment, stakeholder management, and issues management [18]. Environmental assessment describes the examination of the organizational environment to identify and anticipate relevant influences. For the case of stakeholder management, Wood refers to the definition of Freeman as "any group or individual who can affect or is affected by the achievement of the organization's objectives." [35]. Unfortunately, further details on how to identify or manage stakes and claims are not given. Issues management is the last process and entails the monitoring and handling of relevant topics. In contrast to Wartick and Cochran, Wood assorts issues management as a process, since this topic is less oriented towards the performance itself, but rather on the outcome, as defined in the third dimension of the model [18].

(3) The outcomes of corporate behavior describe the observable/assessable part of social performance. Outcomes classify into social impacts describing their direct influence on society, social programs as processes to fulfill specific goals, and social policies as patterns of decision-making within an organization. Thereby social programs and policies are inspired by processes of corporate social responsiveness [18].

The high level of detail and its broad acceptance within the scientific community [36] make this model an appropriate basis for the case of this research. Although the concept originates from a business context, it is feasible for the HEI context as well, since the model shows a highly contextual and organization-dependent degree of integration. Practical application within the systematic screening of the literature requires a customization of Wood's model for the context of this inquiry. Though principles of corporate social responsibility describe the ethical layer behind organizational social performance, this highly contextual issue describes non-observable features [18], which, consequently, cannot be measured. Due to this, the dimension of principles will be excluded from this examination. Hence, the modified model for the examination of management performance on responsibility in HEIs comprises only the dimensions of processes of social responsiveness and outcomes of social behavior (see Table 2)



#### **3. Materials and Methods**

This study applies a systematic review of scientific papers and academic publications, following a structured multi-stage process after Fink and Tranfield et al. [36,37]. The study uses Wood's model for systematically scanning literature on the research question of how HEIs put social performance and responsible management into practice along the dimensions of processes and outcomes of institutional social performance (see Table 2) and follows a three-stage procedure, as described below.

*Stage 1: Selecting Search Terms and Databases*

As the study found no literature issuing the assessment of responsible management or social performance at HEIs on Google Scholar, the structured research process was extended on the electronic databases Ebsco (Academic Search Complete and Business Source Complete) and Web of Science for publications with topics, titles, abstracts, or keywords. A detailed description of the search and the selection of relevant literature can be found in Tables A1–A4.

The search terms and phrases were derived from the objects of investigation, resulting in three thematic blocks: (1) the social responsibility and social performance, with the phrases "sustainab\*" OR "CSR" OR "soc\* resp\*" OR "soc\* perf\*" OR "CSP" OR "SR"; (2) management controls, with the phrases "manag\* control\*" OR "public sector accounting" OR "performance measurement" OR "Public Management" OR "Management Control System\*" OR "MCS" OR "Management Accounting" OR "MAS"; and the (3) institutional background of higher education institutions, with the phrases "high\* education" OR universit\* OR college\* OR campus OR "business school\*" OR "HEI\*" OR "knowledge intensive organi?ation\*". The three blocks were linked with the conjunction "AND" to combine them within a query. The search terms aim to cover a vast spectrum of potentially relevant articles on issues of social performance at HEIs. For this case, the search terms in the field of management controls and institutional anchoring also contain phrases from the accounting sector. The search terms on issues of social performance focus on a more general level. The phrases were grouped around the terms of social performance and social responsibility, which were derived from the model of social performance following Wood, as well as Reiser's definition of responsible management.

The need for a wide view is even more evident as the literature is unclear about whether to apply the term sustainability or social responsibility. Consequently, the research included both keywords. Although the objective revolves around HEIs and not a corporate context, the keywords also include the phrases "CSR" (corporate social responsibility) and "CSP" (corporate social performance) to identify further articles on social sustainability.

According to Tranfield [38], the review searched the documents for full text after applying the search terms, in order to not be too restrictively focused on the bibliographic data.

*Stage 2: Applying Practical Screening Criteria*

To avoid bias within the results, the study included journals, academic journals, and conference contributions published in the English language between the years 1987 and 2019. The study applies the year 1987 as a starting point, since the publication of the Brundtland report marks a turning point in the global debate on responsibility and sustainability. To gain a high number of suitable results, the study uses selection requirements in the screening process to sort out cases without relevance for the research question. Therefore, publications were checked as to whether the title, topic, and abstract fit the objective of responsible management or social performance in the HE sector. Publications with no thematic affiliation, as well as articles beyond the higher education context, were not considered for further screening.

*Stage 3: Application of Methodological Screening Criteria*

To conclude the screening of the literature, the study checked the literature regarding availability. Moreover, the study applied snowballing to identify further articles from the bibliographies, in order to gain a wider spectrum of potentially relevant publications. To specify and structure the findings, a review protocol was generated (Table 3). The categories for the analysis include the following domains:

1. Bibliographic data: year of publication, geographic origin of research, land of publication and journal, authors, and title;

2. Approach of the publication: research design, data collection method, and data analysis method;

3. Issues of social performance: following the structure of Wood's model, focusing on observable patterns (excluding principles of social responsibility) and snowballing and synthesizing the findings.




**Table 3.** *Cont.*

To be assigned to the dimensions of social performance, the statements had to fit into the definition of either the processes of social responsiveness or outcomes of social behavior.

In the case of processes, the structure splits further, into environmental assessment, stakeholder management, and issues management. Furthermore, the dimension of outcomes consists of the divisions between social impacts, social programs, and social policies. The coding was left open deliberately, to cover a large spectrum of publications on social performance and responsible management in HEIs, and to capture perspectives arguing from different theoretical foundations. The coding was undertaken by one member of the research team and double-checked for intercoder reliability within a PhD workshop to avoid bias and suggestive defamation in the results of the analysis. The application of Wood's model for the content analysis delivers useful results on managerial aspects, even with the exclusion of principles due to their non-observable character.

#### **4. Results**

The following section is structured as a bibliographic analysis and a content analysis, inspired mostly by Schaltegger and Wagner [39] and Seuring and Gold [40]. An overview of the procedure is summarized in Table 4.

#### *4.1. Bibliographic Analysis*

By screening full texts under the application of selection requirements and an availability check with the additional screening of sources within the publications, the review finally identified 50 (45 by systematic review, five by snowballing) relevant studies for inquiry. A detailed itemization is provided in the Appendix A.


**Table 4.** Selection process of included studies.

Looking into the distribution of articles over time (see Figure 1), the first appearance of relevant publications on topics of social responsibility in HEIs occurred in the early 2000s, which might indicate an association with the draft of the millennium development goals. This might also be assumed for the surge of publications since the year 2016, after the inception of the sustainable development goals.

**Figure 1.** Distribution of Publications over time.

The majority of articles apply case studies for research design (75%), followed by systematic literature reviews (25%). Looking at the global distribution, the publications' origins crystalize around distinct geographical regions. The main focus of research is located in Europe (45%), followed by North America (27%) and Asia (17%). Looking deeper into the publications with a European origin, the study observes a publication focus in middle and eastern Europe (Germany, Austria, Poland, Latvia, Romania, Slovenia). Apart from that, Spain and the United Kingdom show the highest rate of publications.

Taking a closer look at the journals addressing social issues at HEIs, the majority of articles originate from the "Journal of Cleaner Production" (21%) and the "International Journal of Sustainability in Higher Education" (18%). The publications from both journals do not originate from a special issue. The remaining publications are scattered across various journals, thematically ranging from Management and Strategy to Public Administration and Organization topics. Table 5 provides a list of included journals and their methodological quality according to the German "Verband der Deutschen Hochschullehrer" (VHB) and the "Australian Business Deans Council" (ABDC) as valuable rankings from the Harzing Journal Quality List.




**Table 5.** *Cont.*

In addition to this, the methodological quality of the journals should be examined in more detail. According to Cook and Campbell [41], methodological quality is defined along distinct criteria: statistical conclusion validity, internal validity, construct validity and external validity. The present study follows these criteria and adapts them, as Cepeda and Martin (2005) propose a similar approach for the assessment of case studies [42]. Since a large number of the literature examined is based on case studies, this provides a suitable instrument for the following investigation to assess methodological quality (see Table 6)

Following Cepeda and Martin's approach, methodological quality is composed of internal validity, construct validity, external validity and reliability. These criteria aim at capturing the conceptual framework, research cycle and theory building along the stages of planning, data collection, data analysis and critical analysis [42].

Internal validity examines descriptions of facts and findings, and their internal coherency with the intended objectives.

Construct validity refers to the explicit and detailed description of methods and procedures, including background information on presumed cause and effects.

External validity embraces descriptions, which allows for the assessment of replicability, and evaluates the appropriateness of applied theories and findings.

Reliability evaluates whether the research question is clearly outlined, and the study design is congruent with it.

According to this definition, the following table provides an overview of the methodological quality of reviewed literature.


**Table 6.** Methodological quality criteria in reviewed literature following Cepeda and Martin (2005).

#### *4.2. Content Analysis*

For the analysis of the publications' content, the study applies the model of social performance after Wood. The focus on processes of social responsiveness and outcomes of social behavior make up the core of the analysis. The presence or absence of social performance indicators, especially in the dimension of outcomes of social behavior, allows conclusions to be drawn on the general management of responsibility within HEIs.

Before looking into the dimensions of social performance, the first issue of interest was to determine the terms in use when discussing social performance. Do the publications apply the term 'sustainability' or 'responsibility' to describe the actions of the organizations in their study? The results show that a vast majority (65%) applies the label 'sustainability', whereas 25% apply 'responsibility', when referring to issues of responsible management or social performance. Ten percent of publications apply both terms interchangeably. Among them, seven articles explicitly concern university social responsibility.

For the investigation of processes of social responsiveness, the study adapts the classification process proposed by Wood.

To evaluate the organizational environment, the study distinguishes between the organizational context in which the HEI is embedded and the institutional approach, which outlines how the institution handles responsibility/sustainability.

The external field in which the higher education organization operates is strongly centered around the different responsibilities they face, ranging from a local to a global level. Casarejos et al. [43] provide an overview of the different matters an HEI has to master. A strong factor acting upon the social responsibility of a university is socio-economic pressure on different levels. This includes the influence of corporations through partnerships, legal requirements/regulation from the government, and megatrends like the demographic development or globalization, challenging the organization to act responsibly. International commitments for sustainability/responsibility, complemented by obligations for disclosure or the availability and efficient use of funding, determine the activities of HEIs.

Shifting the focus to the internal perspective of environmental assessment, the study identifies governance and leadership as core aspects forming the center of contemplation. Leadership operates the responsibility for outcomes, the society, and the world as a whole [44], with strategy, mission, vision, and values defining the point of departure for the commitment and participation of the community. This creates the social license to operate for the organization, as Ayala-Rodriguez et al. [29] point out. Management frameworks and patterns for change develop a culture and understanding for conscious acting, as with the institutional orientation between faculty and administration. Marketing and communication of engagement build a bridge from the internal to the external bonding of the institution. Communication tools (e.g., indicators) finally enable the assessment and benchmarking of efforts.

In the case of stakeholder management, the study examines the question of whether stakeholders play a certain role in the management of social responsibility at HEIs and how their interests are managed by the institution.

A central assignment follows a categorization of the groups of students, staff (academic and non-academic), and administrators [12,20,42]. Students are often perceived as the initiators of engagement, following a bottom-up approach, since they have the opportunity to " ... operate outside traditional decision-making systems and their capability to pressure their universities in ways that employees simply cannot" [45,46] or have the power to attract a certain level of attention through their actions as agents of change. However, students have only a limited time of residence at HEIs and their long-term influence is limited. Faculty and staff act as the long-term campus population and therefore dominate long-term changes [47]. A detailed examination of this interest group reveals the managing staff (presidents, faculty leaders, academic directors, or professors) to be major drivers in terms of sustainable development. This confirms the assumption that top-down approaches are decisive for the implementation of responsibility in HEIs.

In the case of managing stakeholders, HEIs apply a broad spectrum of approaches ranging from (nonfinancial) reporting frameworks on ESG providing unilateral information for the assessment of engagement [10] to the provision of action plans proposing multi-stakeholder management processes [48], shared governance [49], or participation [12,46].

The investigation of issues management, the last aspect of processes of social responsiveness, tackles the management of participatory activities within HEIs. An examination of leadership demands in order to identify relevant attributes seems useful here. Besides individual traits like integrity, respect, or courage, the strategic planning and a sustainable leadership development (embracing teaching and mentoring) which imparts self-reflection, and the empowerment of staff, are considered to be crucial for handling stakeholder demands [46,47]. On the side of procedural issues management, a strategic anchoring within long-term visions affecting the institution's policy and targets positively stimulates responsible leadership processes. As one example, the PDCA cycle is mentioned as an appropriate tool [14].

The analysis provides insights into exchange relationships between HEIs and their environment. Understanding these contextual factors creates a more tangible background from which to gather the organizational approaches to acting socially responsibly as directly observable characteristics. With the examination of perceptible outcomes, the study discloses existing practices reflecting indirectly observable processes, which moreover allows for a verification of the internal consistency of the findings.

Determining community outreach is generally perceived as a difficult task [50,51]. The analysis of impacts on social behavior requires a more detailed proceeding which enables an in-depth analysis. Therefore, the analysis of aspects of organizational philanthropy, impacts on society, and their perception/assessment make up the stages of the examination.

HEIs' impacts are strongly tied to their mission to educate responsible citizens by responding to social needs and adding value to society [4]. Communication and collaboration with the surrounding communities can be seen as an opportunity to strengthen the perception that HEIs are a valuable part of the local community, because they improve the quality of life.

Organizational philanthropy describes the institution's efforts to positively influence its environment. Although HEIs play a key role in society, evidence regarding their direct involvement in the surrounding community focuses on rather general positions. Providing social welfare and synergies with local partners from business or government, or positioning the campus as a living laboratory, describe ways in which they positively influence the local community [52].

The assessment of HEIs' social performance follows the application of measures to quantify efforts and results. Various measurement tools for the university context enable a pursuit of these objectives. A variety of tools, ranging from GRI, SSR, AASHE, STARS to the Campus Assessment Framework or the Sustainability Pathways Toolkit, provide ample opportunity for the assessment and communication of engagement and community outreach. In terms of distinct tools for the measurement of social performance, the possibilities are more limited [13]. Since most indicators are embedded in an overarching sustainability assessment scheme, environmental aspects dominate reporting standards or are prioritized. Current indicators firmly referring to social performance and responsible management include research compliance (e.g., anti-corruption, respect of privacy or outcome-responsibility), human resource management topics (health, training, diversity, satisfaction), human rights (anti-discrimination, child and forced labor), or education issues (curriculum content, teaching quality, graduation rates).

The examination shows that, so far, no reporting tool serves as a standard for assessing social performance. In the context of assessment tools, (environmental) sustainability issues seem to be a managerial fig leaf for HEIs to shrink from their (social) responsibility.

Notwithstanding, even without assessment tools of social performance, it is possible to be sustainable and act responsibly as an institution of higher education [11].

The social programs of HEIs describe the process perspective in the pursuit of certain goals. For a detailed examination, the research on this topic is divided into social (performance) objectives and management perspectives of processes, to enable a better overview of the field.

In terms of organizational responsibility, major targets include issuing holistic approaches fostering quality management or society outreach to satisfy stakeholder needs and tackle social problems [19,50]. As there is a lack of mission statements embracing responsibility issues [53,54], the use of blanket statements [51] fostering the public image of the organization [53] suggests the need for more responsible management. The evolution of mission statements simultaneously calls for an investigation of managerial processes [55] to achieve better governance and leadership [48]. As a driver of institutionalization, processes depend on the selection of objectives before initiating collaborative structures fostering commitment and comprehension [7] in order to overcome barriers and trade-offs [56]. These integrative efforts lead to an improved understanding and organizational culture of holistic and systematic change [44].

As the last aspect, social policies describe procedures concerning social issues at HEIs, as well as their implementation. The examination shows certain processes often begin in the course of informal activities [20] leading to an incremental incorporation [7], flanked by participatory processes, management frameworks, or guidelines, and driven by top level support, clear objectives, monitoring, and an institutional framing [7]. Mission statements [54] defining a certain policy [57] with transparent objectives, clear processes, monitoring, and communication patterns [58] promote a culture of social responsibility [7]. Human Resource Management functions as a critical success factor by providing the necessary resources [16] and spurring organizational change and commitment.

#### **5. Discussion and Conclusions**

The present study examines the social performance of HEIs and the operationalization of responsibility management, and answers the question of how HEIs' social performance management can be quantified within the dimensions of social performance following Wood [18]. The systematic literature review gives insight into the status quo of responsible management practices of HEIs and investigates HEIs' social performance along the dimensions of processes of social responsiveness and outcomes of social behavior. The study examines a structured approach to management activities, which makes it feasible to exercise steering and assessment processes. The results show social performance is an issue of HEIs' responsible management, though the forms and distribution across the examined dimensions vary.

The present study contributes to the discussion on management performance on social issues and responsible management at higher education institutions in three ways.

First, the study provides a systematic overview of different research approaches to responsible management affecting the social performance of HEIs. The study shows that, regarding the management of social issues, there is no common practice. It becomes evident that HEIs predominantly focus their engagement on processes of social responsiveness, especially environmental assessment and stakeholder concerns. This shows a general willingness to perceive social responsibility and act accordingly, but falls short regarding outcomes, especially in the case of determining impact or transferring commitments into actions. This can be explained by the generally low hurdles for HEI management to produce statements on voluntary obligations. Taking action, though, by implementing concrete structures and processes, appears to be of minor interest, since this step has an obligatory character as it requires the allocation of distinct resources with only loosely predictable long-term outcomes. The low proliferation of responsibility management structures and processes suggests a weak institutionalization, which might result from vague objectives, which inhibit to achieve certain goals. Since environmental assessment and stakeholder management appear to be well-elaborated topics, HEIs seem to suffer from a lack of translation mechanisms to integrate these issues into their responsible management.

Second, the study expands on the current literature on responsible management in HEIs by exploring the role of management performance for the implementation and steering of social issues. The application of social performance measures opens up new perspectives for the understanding of aims and conditions of a successful implementation of managing social issues within HEIs. The abundance of performance information on the features of environmental assessment and stakeholder management, contrasted with the weak expressions of structures and processes, illustrate the need for a systematic management approach. Though the willingness to shoulder social responsibility values is recognizable, implementation so far is limited to a measurement of outcomes.

Third, the large variety of case-based research underlines the demand for a structured approach. The non-standardized use of the terms 'sustainability' and 'responsibility' in the examination on social performance issues opens a general discussion on whether 'responsibility' and 'sustainability' should be treated as complementary or competing terms [56]. Moreover, the subsumption of instances, and of responsibility, under the term 'sustainability', is misleading when talking only about social performance. The versatility of the topic makes it occasionally difficult to assess and manage all dimensions under one management or to supply responsibility from a single source (e.g., different stakeholder claims, functional overlaps of different institutional units and responsibilities).

Based on these findings, the study proposes implications for decision-makers and researchers. First, the assessment of the literature reveals a recent interest in social matters, responsible management issues, strategy implementation, and leadership, as well as the measurement of outcomes at HEIs. So far, ecological sustainability prevails within the scientific discourse and reporting, whereas social (performance) issues play a minor role. Following the triple bottom line, the role of social sustainability has to be emphasized and strengthened in order to be more balanced alongside environmental sustainability. Therefore, a consistent wording and a clear definition of both concepts are crucial to gaining and improving consistency. The synonymous use of the terms 'responsibility' and 'sustainability' mirrors the goal of a harmonization of both terms. The current underrepresentation of social performance issues in HEI management, caused by an unclear ex-ante assessment of consequences entailing voluntary commitment with incalculable future obligations, must be overcome by a clearly defined management construct.

Second, HEIs bear a special responsibility towards their staff members. Besides the fact that the staff constitutes the long-term campus population, their role is often given minor importance compared to other stakeholder groups. The study discovers that participatory approaches and shared governance formats can create a proper framework for the implementation of responsible management affecting social performance. Voluntary engagement has the potential to initiate activities, which can later be transferred into formalized structures and processes. This has the potential to maintain interaction with stakeholders in an ongoing process that reflexively reviews the university's efforts to improve social impacts.

Third, the examination of institutional impacts of universities' social performance provided weak evidence for the application of appropriate tools, as well as for long-term measurements, which deliver a satisfying assertion on engagement. A missing definition might be cause for the diverse understanding of responsible management, which results in the relatively weak impact of actions. So far, existing assessment tools for sustainability at HEIs provide unsatisfactory results.

As in many other surveys, the present study also shows different limitations, providing future research needs. As of now, the impact of social performance seems to be an irrelevant measure for HEIs, since existing assessment tools marginalize social topics [13]. Nevertheless, it is important to mention that a voluntary engagement for social responsibility can pursue various pathways through an organization, even without proper measures. Formal, non-formal, and informal institutionalization efforts can raise new possibilities for further research, especially by differentiating the roles of various stakeholder groups and the meaning of contextual factors for possible implementation strings. In that context, it should be questioned whether principles of social responsibility are in fact non-observable features of an organization or if it is possible to redefine and identify them based on changing practical parameters, communication, and transparency. This would also be useful for a definition of the term 'social responsibility'. Since social responsibility is a diverse field in the institutional context and finding a generally accepted definition is challenging, a proper solution finally has to emerge from the communities negotiating the field of tension between sustainability and responsibility [32]. Along these implications, the research leaves open the question as to which social responsibilities HEIs actually bear in order to fulfill their license to operate as social institutions. Since the study shows that the social impacts of organizational activities are difficult to quantify, a major task for future research will also be to find proper measurement methods with which to solve the question of how HEIs can pursue purpose in line with acting responsibly, and which values should guide them along the way. Further studies could also take up some parts of the results to provide a more significant contribution to the field of responsible management of HEIs. In the end, the major question is whether HEIs will manage to walk the talk, or continue to simply provide lip service regarding their social responsibility.

In short, this paper highlights a model that, by providing accessibility through understanding, enables the active managing of HEIs' social responsibility. The systematic review of the literature shows that HEIs are strongly determined by their mission on research and teaching, which, so far, exceeds other external demands from outside the organization. The strong orientation toward internal affairs emphasizes governance and management topics as relevant influencing variables determining the organizational focus. As a result, HEIs are currently in the process of deepening their engagement with themselves and their mission statement, and forming clearly defined goals, which reflect their social responsibility and have a meaningful effect. Due to a lack of urgency and external pressures, the search for social responsibility beyond research and teaching is difficult. Governance structures and management processes can have a steering effect, in order to proactively shape responsibility and bring it into society along transparent processes.

The implementation of performance measures on social responsibility in HEIs allows their integration into general performance measurement, which strengthens the interconnectedness between economic, environmental, and social issues. Realizing these potentials enables an effective management of social issues and encourages responsible habits, even in fields that have so far evaded measurement (e.g., principles of corporate social responsibility). To become effective operationally, responsible management has to harmonize social performance principles, processes, and outcomes, to create more rooted solutions which balance local and global demands. Managing social performance in this way can help HEIs to close the gap between social values and organizational behavior, and thereby improve their social legitimacy.

**Author Contributions:** Conceptualization, N.R.; methodology, N.R.; software N.R.; validation, N.R.; formal analysis N.R.; resources, N.R; data curation, N.R.; writing—original draft preparation, N.R; writing—review and editing, N.R.; visualization, N.R.; supervision, N.R.; project administration, N.R.

**Acknowledgments:** Open Access Funding by the Publication Fund of the TU Dresden.

**Conflicts of Interest:** The author declares no conflict of interest.

#### **Appendix A**


#### **Table A1.** Search on EBSCO database.



#### **Table A3.** Search on Web of Science database.


**Table A4.** Steps on filtering relevant literature on Web of Science search


#### **References**


© 2019 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **Sustainability Assessment and Benchmarking in Higher Education Institutions—A Critical Reflection**

**Sandra Caeiro 1,2,\* , Leyla Angélica Sandoval Hamón <sup>3</sup> , Rute Martins <sup>1</sup> and Cecilia Elizabeth Bayas Aldaz <sup>3</sup>**


Received: 13 November 2019; Accepted: 7 January 2020; Published: 10 January 2020

**Abstract:** Higher Education Institutions (HEIs) play a crucial role in implementing practices for Education for Sustainable Development (ESD). This implementation should be done in different dimensions according to a holistic and whole-school approach. Different tools have been adapted and developed to assess this integrated approach. The aim of this research is to critically reflect the existing tools to assess and benchmark ESD implementation and to discuss their applicability in two case studies. Two public Universities in Southern Europe, with headquarters in the capitals of Portugal and Spain were selected to assess and compare the integration of ESD according to a whole-school approach—Universidade Aberta in Portugal and Universidad Autónoma de Madrid in Spain. After a critical analysis of the existing tools based on literature review and a list of criteria classified by experts, two tools were selected to be applied in the case studies. The online Sustainability Tracking, Assessment & Rating System Reporting Tool was used in Universidade Aberta and Green Metrics tool was used in Universidad Autónoma de Madrid. The tools were complemented with focus group with key-actors in both universities. The results obtained allowed to identify the need to define a common objective of the assessment tools and limitations they still have. The tools need improvements on their development namely to integrate the external impact of Higher Education Institutions on sustainability, to integrate participatory processes and to assess non-traditional aspects of sustainability. This research hopes to contribute to the continuous research about the usefulness of these assessment and benchmarking tools as drivers to HEIs improve their sustainability performance and their role as agents of changes.

**Keywords:** higher education institutions; sustainability benchmarking; sustainability tracking; assessment and rating systemTM; green metrics; education for sustainability

#### **1. Introduction**

Higher Education Institutions (HEIs) have a critical responsibility in education for sustainable development (ESD) due to "the main component for raising awareness of SD among the population" [1]. In other words, ESD provides the knowledge and skills for students to start to create SD initiatives. Furthermore, HEIs should "lead by example" [2].

Sustainable development as a development model integrates environmental, social and economic considerations [3] and so HEIs must assume a holistic focus in all activities. Indeed, HEI with or without governmental policies and recommendations are moving towards holistic and systemic approaches when addressing Education for Sustainable Development. The Sustainable Development Goals (SDG), recently adopted by United Nations in 2015 [4] and in particular, SDG 4, are an additional driver for the implementation of sustainability at HEI in an integrative way.

According to the United Nations guidelines followed by several researchers the integrative approach to implement sustainability in HEIs includes six major ESD dimensions to allow a whole-school approach—(i) Facilities or Operations; (ii) Teaching and Curriculum; (iii) Organizational Management; (iv) External Community; (v) Research; (vi) Assessment and Communication (based on [5,6]).

Based on early work [7,8], some authors stress that the integration of sustainability in HEIs can be done at various levels, from national to the institution level [9]. The institutions that integrate sustainability according to the whole-school approach (in terms of dimensions and involvement of the whole HEIs community) can achieve the sustainability maturity curve, thus enabling them to be agents of change and transformation. In this context, Kapitulcinová [9] introduced the model from a "business-as-usual university" to a "sustainable university" where sustainability has been fullyintegrated under three degrees, initiation/awakening, implementation/pioneering and institutionalization/transformation, including in this last stage a consolidation of changes. Thus, the HEIs are under mounting pressure to partner with societal stakeholders and organizations to collaboratively create and implement sustainability-advancing knowledge, tools and societal transformations [10]. So, in recent years, an increasing number of institutions have begun the adjustment and restructuring of education, research, campus operations and community outreach towards sustainability [8,11,12]. However, deep and full integration of sustainable development at all of dimensions and community of HEIs is still lacking [13,14].

Different tools have been developed to assess and benchmark ESD implementation at HEIs but how well are being performed in case studies and how well they evaluate sustainability and its impact is still an open question. This study attempts to provide a holistic sustainability maturation path reflecting essential dimensions that HEIs need to approach and assess as whole-school integration towards a potential sustainability management change. The aim of this research is, in the first part, to critically analyze the existing tools to assess and benchmark ESD implementation in HEIs. In the second part of the article, the aim is to assess the integration of ESD according to a whole-school approach through the use of those tools in the case studies and discuss their applicability. Two of the analyzed tools were selected and applied in two HEIs in Portugal and Spain. The assessment was complemented with stakeholder's engagement. European HEIs have been ahead in the implementation of sustainability at the different dimensions [13,15]. In particular, in southern Europe neighbor countries like Portugal and Spain, despite the lack of national policies, HEIs are working towards ESD implementation through the development of plans and actions in sectorial areas. The two case studies are public Universities (one smaller and in distance learning regime and others in traditional learning and bigger) engaged in the last years in integrating sustainability in different dimensions. They have their headquarters in the capitals of Portugal and Spain and are classified in levels of excellence in their teaching and learning methods: Universidade Aberta, Portugal and Universidad Autónoma de Madrid in Spain.

This article starts after this Introduction with a literature review about the tools to assess sustainability in HEI (Section 2). Section 3 presents the methods for the tool's assessment and case studies evaluation. Section 4 describes the case studies, Universidade Aberta and Universidad Autónoma de Madrid. Section 5 discuss the results and the final section is dedicated to final conclusions and contribution to new knowledge and research implications.

#### **2. Sustainability Assessment in Higher Education**

Given the rapid growth of ESD initiatives at HEIs, the measures, assessment and reports of HEIs progress toward SDG has become increasingly important [16]. The assessment of sustainability in HEIs is one of the most important dimensions of EDS implementation in HEIs and could be conducted base on specific tools, that allows to assess whether all possible dimensions to the implementation of sustainability are being implemented and whether they are doing so holistically [9]. Also, when well-developed these tools can be used as benchmarking practices, comparing HEIs processes and performance metrics. Nevertheless, these tools must be able to uniformly evaluate the implementation of sustainability in HEIs

without necessarily placing them in a ranking or competition. So, the tools must identify the important themes, be measurable and comparable, go beyond eco-efficiency, measure progress and motivations and be understandable to a broad set of key actors [17]. According to other authors [18], these tools can be based on indicators and conceptual models that support sustainability decisions, as well as facilitating communication efficiently and for a wide audience, knowing how to respond to complex processes capable of assessing the transformation for sustainability.

Much work has been done on the development of tools to assess sustainability specifically in HEIs, showing the importance of the theme. Several articles have reviewed these tools from different perspectives (e.g., [7,9,14,16,17,19–33]). However, it is observed that in these various reviews, the systematization is not homogeneous, noting a lack of common designations and objectives and including tools that do not have as main objective the assessment of the implementation of ESD or are not per se an assessment tool. So, for example, the reviewed tools:


Thus, for this study a systematic review of the tool for sustainability implementation assessment in HEIs was conducted, based on the following conditions:


Based on these criteria 27 tools were searched on google scholar using the key-words: "Sustainability assessment" and "Higher Education," from October 2018 and March 2019. Each tool was then characterized (see Table 1).


**Table 1.** Sustainability Assessment Tools in Higher Education (HE) and brief description.






According to several authors, the overall implementation of these tools is still low and its development is still at an early stage [14,28,32].

The various tools for assessing the sustainability of HEIs are mostly based on indicators, using graphs or final rankings to communicate the results. Indicator-based tools have the advantage of being potentially more transparent, consistent and comparable, thus useful for monitoring and decision support [14,18], although support for decision making is not yet fully demonstrated [65].

Another common characteristic of these tools listed in Table 1 is the fact that they are filled out by self-assessment, requiring only a leader or researcher to complete them. In order to create a sustainable university, it is important not only to use assessment tools for a real application as well as integrate on the process different agents of Higher Education Institutions [32]. Stakeholder participatory approaches can be seen as a requirement, as well as a benefit towards the integration of SD into the university culture [66]. Furthermore, active stakeholder participation is essential to grow the model's level of complexity, promote model ownership and use it in the organizational strategic planning process in a higher education organization [67]. Stakeholder engagement is also crucial to achieve the visions and goals for a Green university [68] and current SDG. The second draft of the People's Sustainability Treaty on Higher Education [69] divided higher education stakeholders into three broad categories: (1) those engaged in the activities of higher education institutions: executive, academic managers, educators, researchers and students. (2) those engaged in the higher education system: administrative officers, ministries, assessment bodies, international organizations. (3) those forming part of the communities, which the HEI system serves: local communities, professional bodies, companies, among others. Hence, a socially responsible HEI considers stakeholder behavior and perception to better understand their expectations and priorities and use these to define the strategy and goals, to monitor the objectives in view of promoting activities and accountability and to enhance a community-university engagement. At the end it contributes to change management and to a more mature sustainable university [9]. Thus, sustainability reflects a condition based on the relationship between stakeholders and HEI [70].

Based on the characteristics of these tools (see Table 1), the 27 tools were critically analyzed to evaluate their real assessment of the sustainability implementation and integration in HEIs (see the sections of the methods and results).

#### **3. Cases Studies**

The Universidade Aberta (UAb) was founded in 1988 with the distinctive feature of being the only Portuguese public distance education university, a distinction that still remains today. Filling the Universidade Aberta vision and mission of being a global university, the Universidade Aberta offers undergraduate and graduate higher education courses and Lifelong Learning courses, which are especially dedicated to the whole Portuguese speaking country community. The Universidade Aberta campus corresponds to four facilities, namely, its headquarters in Lisbon and two other support buildings in Coimbra and Porto (regional offices). The Universidade Aberta also has other facilities in Lisbon and Local Learning Centers spread throughout the country but since they are not owned but rented, they were not considered in the sustainability evaluation. In 2018 Universidade Aberta community comprise the rounded numbers of 6000, which 5000 were full-time students and exclusively engaged in distance education and 340 employees, of whom 150 belongs to the academic and research staff and 190 to the administrative staff. It is structured in 5 academic departments: Science and Technology, Social and Management Sciences, Distance Learning and Humanities, Lifelong Learning, with an educational offer of 10 graduate Ph.D. programs; 22 graduated master programs, 11 under graduated programs and 9 post-graduated programs. UAb has 2 research institutes and 5 more research institutes with branches in this university.

The Universidade Aberta has focused on the quality of its service and has been distinguished by several national and international entities: (a) EFQUEL Award—European Foundation for Quality in E-learning in 2010; (b) the UNIQUe—The Quality Label for the use of ICT in Higher Education (Universities and Institutes) in 2010; (c) the 1st Level of Excellence Committed to Excellence (c2e2) of the European Foundation for Quality Management (EFQM) in 2011; (d) 2nd Level of Excellence Recognized for Excellence (R4E) of the European Foundation for Quality Management (EFQM) in 2016; (e) the certification of the International Standard Organization (ISO) 27001 by the Portuguese Association of Certification attesting the security of its platform of e-learning and ISO 9001 quality, in 2017. UAb has been applied sustainability in different ways, namely: (i) curricula (through a e-learning three cycle degrees system from undergraduate, to master and Ph.D. aiming to actively promote education for sustainable development, along with an increase in transdisciplinary across subjects and also through non formal courses about Climate Change, Education for Sustainability and Environment awareness), (ii) application of a quality management policy with a Recognition of Excellence, (iii) a specific inclusion program for students with disabilities. UAb formal compromise to Sustainability was achieved by being an institutional member of the Association for the Advancement of Sustainability in Higher Education (AASHE) and a signatory member of the Letter of Commitment for Sustainable Campus in Portugal, both in 2019. UAb has no formal office for Sustainability due to its small size but sustainability issues are informally addressed within the Quality Office.

The Universidad Autónoma de Madrid (UAM) is a Spanish public university established in 1968. In 2018, 26.733 students of all levels were enrolled at the UAM. This university has a teaching staff and/or researcher around 3.141 and an administrative and services staff around 1.053. The university is organized into eight schools: Sciences (biology, mathematics, physics and chemistry), Economics and Business, Law, Computer Science and Engineering, Arts (philosophy, history, philology, translation and interpretation), Education and Psychology, offering a wide range of programs in different scientific and technical fields and in the Humanities. UAM has 11 research institutes and these are located on campus, as well as the Madrid Science Park, with growing university-business collaboration (contracts, internships and sponsored chairs). In 2009, UAM was declared International Excellence Campus, at the same time with the Spanish Research Council (CSIC). Nowadays, UAM has been placed among the top universities for its levels of excellence in national and international rankings. In the QS World University Ranking 2019, the UAM has managed to locate itself in the first position in Spain and in the 159th place in the world.

Most of the faculties and specialized institutes are on the Cantoblanco campus, 15 km North of Madrid. The Faculty of Medicine is on another campus (near La Paz Hospital). These two campuses (Cantoblanco and La Paz) were considered in the sustainability evaluation. In the field of sustainability, this university, from the Rio de Janeiro Summit (1992) (where a global action plan for Sustainable Development was approved: Agenda 21), formalized its commitment to Agenda 21 through the ECOCAMPUS project. In 1997, the Ecocampus office was created, which has a special involvement in the maintenance of the campus as a sustainable territory and at the same time promotes activities related generally to sustainability. Social commitment and sustainability continue to be part of the frame of reference of the different lines of action for the entire institution in the 2025 strategy. UAM has also an SDG Lab that is a multi-stakeholder initiative that contributes to the implementation of the Sustainable Development Goals at the UAM.

#### **4. Materials and Methods**

#### *4.1. Tools Assessment*

The critical analysis of the tools listed in Table 1 was based on earlier reviews and according to the following criteria (based on the research of [9,18,24,28,65]: (i) comprehensibility; (ii) comparability; (iii) availability of baseline data; (iv) assessment of progress over time; (v) comprehensiveness and integration of sustainability dimensions in HEIs; (vi) usefulness for decision-making and communication; (vii) level of participation of the public or key actors and (viii) tool accessibility on the internet. The tools were then classified from 1 to 3 for each of the criteria (1. Low, 2. Medium and 3. High), based on documentary analysis of the tools and expert knowledge (according to the methodology referred in Reference [71]).

The classification of each tool was conducted independently by four judges (authors of this paper) using the defined criteria. The procedure proposed in Reference [72] has been followed in order to determine the level of agreement between the judges (values between 0 and 1 for each agreement between two judges):

Index of Agreement = (C1,2 + C1,3 + C1,4 + C2,3 + C2,4 + C3,4)/6

For data analysis, the mode and relative frequencies were calculated for each criterion. An average of the Index of the agreement was calculated for each criteria and tool. Two of the listed tools were not classified since their information was not available in English (DUK was only accessible in German and ASSC only accessible in Japanese).

Validity, reliability and generalizability are limitations associated with this type of qualitative approach [71] and were weighed up in the qualitative assessment and discussion of the results and when drawing the conclusions.

#### *4.2. Sustainability Assessment in the Case Studies*

The sustainability assessment process at the Universidade Aberta and the Universidad Autónoma de Madrid was carried out using different tools, STARS and GM, respectively, complemented with stakeholder's engagement (focus groups/workshops) in each University. The focus groups and workshops were used for engagement and awareness of the assessment process and also to find paths of improvement (Table 2). The STARS and GM tools were selected since they were well classified according to the main criteria and also in terms of free access on the internet (see Section 5.1 heading the in results section).

For UAb, the implementation of the sustainability assessment was conducted across the year 2018, corresponding to a 3-year assessment analysis (2015 to 2017). STARS 2.1 version tool is based 5 dimensions with different weights each: Academic and Research (20%), Involvement of Key Actors (20%), Campus Operations (35%), Planning and Administration (15%), Innovation and Leadership (2%) and a total 74 indicators divided in each category. The indicators are quantified and filled in a web application with a written justification or document upload. The different indicators and methodological procedures are explained in AASHE [58]. The raw data collection was the first procedure, involving not only a web search as also the requirement of specific informants for the technical information, reach by a face to face interview or information request by email. Possessing the requested data, the STARS assessment was fulfilled, resulting in a diagnostic report on the implementation of sustainability at the Universidade Aberta, where the most and least punctuated indicators were identified, revealing the weaknesses and the strengths to the implementation of sustainability. Since UAb is an institutional member of AASHE, a final assessment of the STARS scores was conducted, through an internal (by the rectorate) and external validation (by the AASHE technical staff) to allow a final awarded rating between bronze, silver or gold label.

With the final STARS report information, two focus groups (one with university experts and others with different stakeholders from the all University—see Table 2) were held in June and July of 2018. The main objective of the participatory moments was to show and discuss the STARS results and development of proposals for improvement for the implementation of sustainability at the Universidade Aberta. In a first part of the focus groups the information provided in the STARS report was presented and improvements were then proposed according to STARS dimensions. The first focus group was conducted with university experts that have been working in UAb sustainability implementation, including the Vice-rector for quality (since the University does not have any green sustainability office and three researchers or teachers). The other focus group was organized with different stakeholders from the all University from the different departments (one professor per department chosen according to a convenience sampling), administrative staff (leader of each administrative service) and two students from environmental and sustainability graduation programs (chosen according to a convenience sampling). Detailed information about all the methods applied in UAb available in Reference [73]. These discussion groups were the first time the university stakeholders participated in discussions related to sustainability.


**Table 2.** Operationalization design of the sustainability assessment in the case studies.

The process of evaluating the implementation of sustainability in the UAM has begun in 2013 and annually assessed (from 2014 to 2018), through collecting the data according to the six dimensions stipulated by GM with different weights each: Setting and Infrastructure (SI) (15%), Setting and Infrastructure (SI) (15%), Waste (WS) (18%), Water (WR) (10%), Transportation (TR) (18%), Education and Research (ED) (18%). All information about the indicators and methodologies are available at Green Metric [50]. The indicators are quantified and filled in a web application (survey type). Each year in a predefined calendar, Green Metrics validates the submitted universities and publishes the international rating scores.

Representatives of the various groups of the university have participated in the contribution of these data: administration and services staff, teacher and manager (see Table 2). They participated through face-to-face interviews and email, together with collecting the available information, for example in corporate reports and on the web page of the University. The GM assessment along the years allowed to observe the evolution of the university in each of the dimensions and to compare with the scores of participating universities worldwide in GM.

For a review of the sustainability implementation and improvement proposals of the university sustainability implementation, two participatory techniques were carried out, focus group and workshop) (see Table 2). The focus group and workshop were then conducted using semi-structured questions based on the four dimensions of STARS tools to allow better comparison with UAb case study (Academics; Engagement, Operations and Planning & Administration). The collection process was in May 2018. Participants were selected based on their crucial role in the university management system, specifically on the sustainability activities at the campus. In the 1st focus group, 4 members managers of the Eco-campus team participated and in the 2nd workshop the 3 Vice-Chancellors of UAM with responsibilities in the sustainability issues were called to participate. In the 3rd focus group, 6 students' leaders from the Faculty of Business and Economic Sciences participated in the participative process (according to convenience sampling).

In a fourth participatory phase, SDGs LAB—Sustainable Development Goals Workshop was held. The UAM SDGs LAB was organized by the vice-chancellor of Sustainability and Campus and coordinated with professors and researchers in the field and 6 external experts from the town hall staff, already used to collaborate with the university, in a total number of 22. It took place in the UAM Campus for three days. It was the first participatory workshop to design a roadmap to improve the contribution of the university towards SDGs. This workshop was organized in three stages: (I) Inspiration: review of previous experiences and new inspiring ideas, (II) Observation: potential contribution to the campus in situ and proposals, (III) Implementation: discussion, analysis and outcome.

The results of the discussion groups in UAb and UAM were transcribed and then manually analyzed to identify recurring topics in the responses according to Reference [71]. The analyses were validated by the participants.

#### **5. Results and Discussion**

#### *5.1. Tools Assessment*

In an overall performance, Comprehensibility and Comparability were the criteria higher ranked by the judges (72% and 56% respectively classified as 3, see Table 3). Since the majority of the assessed tools are based on indicators (see Table 1) it is expected that they fulfil these criteria, as well as being a support tool for communication and decision-maker. Nevertheless, is still in doubt of real demonstration if indicators can indeed support decision-maker (only 48% were classified with the higher score in this criterion and also the index of agreement between judges was low—43%). This doubt is in accordance to Ramos [65] in his reflection article. Many of the assessed tools are adaptations to existing ones, often being used to contain geographic specificities (namely from North America, Latina America, Asia and Europe), what make the comparison at the national level easier but more difficult at the international level.


*Sustainability* **2020** , *12*, 543


#### *Sustainability* **2020** , *12*, 543



 (1)—Relative Frequency (1) low; (2)—Relative Frequency (2) medium; (3) 

#### *Sustainability* **2020** , *12*, 543

The more complex and complete the tools become, the more complex become to fill them and so the access to the basic data needed (only 32% of the analyzed tools had a maximum data availability rating, such as tools where is only need to answer "yes" or "no" or in the case of multiple-choice closed answers).

The heterogeneity of dimensions of sustainability implementation in HEIs covered by the tools is still remarkable, with the teaching and curriculum dimension and campus operations remaining the best addressed. Also, the environmental pillar is the more addressed, neglecting the social and economic pillars. These results are in accordance with [74], who also found in a study conducted in Spanish Universities that more attention is given to the environmental pillar and that is still necessary to achieve an integrated perspective of sustainability in universities. Those results justify the maximum ranking of only 36% of the tools in the criterion of coverage of sustainability implementation dimension and the same percentage in the criterion of measuring the progress of sustainability implementation in HEIs (see Table 3). Assessment of the progress over time was also the criterion with a higher level of disagreement between the judges (35%). According to Reference [75], variability between judges' classification is usually expected. Nevertheless, this criterion can be subjective since the assessment along the years can also depend on the HEIs and not only the tool himself. These results are in line with recent studies (e.g., [14,16,32]). Also related to this criterion, the state of development of the tools is revealed by the lack of assessment of impact outside the institution, that means on society in general and long-term impact [76].

The Participation and Accessibility criteria were the ones that have the lowest classification (12% and 24% respectively classified as 3, see Table 3). Concerning the level of agreement between the judges, Accessibility was the criterion that reunited more consensus what is easily explained by the fact that the tool is or not available on the internet to be used (Index of agreement of 77%). Indeed, only a small number of tools have an easily accessible application on the internet to fill in the data and obtain the final result. Participatory approaches have gained increasing attention in the implementation of sustainability in higher education but often remain vague and less addressed in sustainability assessment procedures, as Disterhelft et al. [66] also defended. The policy agenda of ESD calls for innovative and more transformative approaches than reductionist practices, in order to respond better to the need for an institutional learning culture that envisions dialogue and change as stressed by the same authors.

The results of the classification of the tools by the judges (Table 3) highlighted that STARS and USAT have good performance in 6 criteria ranked with maximum classification (both had a sum of 22 points in a maximum of 24). More specifically, STARS ranked medium criteria only in Data access and Participation categories and USAT in Participation and Accessibility criteria. Additionally, from Berzosa et al. [32] experience of "USAT is simple to apply, however, complicates comparations and benchmarking, as there are not any mechanisms to standardize the interviewee and the evaluator."

Following these tools were GASU, GM, SUM, SLS and GP (a sum of 20 points, see Table 3) with 5 criteria with the highest classification. For example, GASU has being pointed out by other authors (e.g., [7,16,29]) has having the advantages of giving institution visual illustration of sustainability, turning easy to compare and contracts the universities efforts towards sustainability within and among other universities. Also, AISHE was well classified with a sum of 20 points, with 4 criteria ranked with maximum classification (and a high index of agreement between the judges—71%). As discussed by other authors like Berzosa et al. [32] "AISHE score depends on stages of development of sustainability policies and actions within the organization and varies from an activity oriented (1) to a society oriented (5) approach." In addition, according to Alghamdi et al. [16], AISHE was designed to incorporate only the most significant criteria and not necessarily the whole framework.

Regarding the tools that scored the lowest on the ranking, TUR, HE21 and GMID were the ones with 3 criteria with the lowest classification, where Comparability and Accessibility had poor scores in all three (see Table 3). HE21 [31] is difficult to measure and compare, also indicators may not cover most important issues. ESDGD, P&P and TUR are the tools that had a high divergence in the judgment agreement (44%, 50% and 42% respectively), evidencing a great variability in Understanding, Comparability, Progress Over Time and Support to Decision criteria. TUR and GMID are old tools

only available in research papers. P&P, HE21 and ESDGD are tools adapted specifically to be applied in Universities in a geographical region (UK).

Considering the confounding results, THE and GM evidenced a disperse result for the Participation and the Support to Decision criteria, respectively, because the score was divided in the lowest and highest ranking (also with a low index of agreement between judges 50 and 56%, respectively). In GMID, there is a clear outlier in the Accessibility category (since the tool is not available online) and in THE, an outlier for the Comparability category (since it is an international ranking). These facts highlight the importance of the classification being made by several judges and reducing the bias of the results.

Overall, the results confirmed STARS as the tool that collected a high level of agreement among the judges (71%), with the four criteria expressing the same score, Understanding, Comparability, Sustainability Broadness, Support to Decision. Nonetheless, STARS major strength according to Sonetti et al. [31] is its preference for performance over strategy. Earlier studies that reviewed these kinds of tools highlighted that STARS, AISHE and SAQ have a higher incidence on the percentage of indicators for the Governance (in accordance with reference [25]) and Operations, as highlighted by Reference [28] dimensions. However, STARS has the widest coverage across all indicators, capturing a little of all areas compared, for example, with AISHE and SAQ [as also defended by other authors [24,28], SRC (in accordance with Shi and Lai [77]) or with GM [also as highlighted by Lauder et al. [49]. In addition, as authors like Berzosa et al. [32] claimed "the main weaknesses of SAQ are those related with open-ended questions, not establishing a final score so it is difficult to apply it as a tracking tool."

In a more recent study, STARS appear to be comparable to AISHE and BIQ-AUA, considering the availability of academic and management staff as agents of change in the implementation of sustainability within universities [9]. Analyzing the filing process between STARS and AISHE, it can be seen that the former has a higher percentage of closed questions [24], so it is possible to deduce a greater ease of completeness and greater reliability when comparing the results, thus proving to be more efficient for a regular implementation [14,26,73]. This information is also underlined in a comparative study of STARS with other tools [78]. Authors such as Stough et al. [79] highlight various strengths in STARS namely innovation, understanding, popularity and be based on a holistic and integrative approach to sustainability, while also considering the United Nations Development Goals [79].

GM tool was also overall well classified, besides worst classification in terms of Participation and Progress Over Time criteria but has been also widely used since is free, easy to fill and good as a benchmark between universities worldwide, as stressed by many other authors (e.g., [16,49,74]). As Lauder et al. [49] stressed under scientific analysis, no ranking can be free from at least some limitations, resulting in unavoidable practical considerations, such as the need for the ranking to be at a level of complexity that can appeal to a wide audience. In addition, Sonetti et al. [31] mentioned one of the major weakness of GM is the use of generic quantitative indicators which does not underpin local dimensions as well as lack of the social dimension.

#### *5.2. Application of the Tools in the Case Studies*

Universidade Aberta scores for each STARS dimensions are shown in Table 4 and Figure 1 for the period of 2015–2017. The overall scores allowed to awarded the University with a Bronze label on April 5th of 2019 after a three-time external review process by STARS technical staff where some scores rectifications were made after documentation and numbers checked. UAb STARS report is now listed on the website of AASHE. Academic and Research dimensions are the ones where UAb is better classified leaving much room for implementing sustainability in its Planning and Management dimension, unlike in other universities where the focus is more on the Campus Operations [80]. Since UAb is a distance learning institution there is no formal campus for students' classes and the resources demanding are low (in particular in terms of energy) what justifies that difference. Nevertheless, Campus Operations improvements can be put in place as discussed in the focus groups (see Table 4). However, it is worth mentioning the importance of this approach for recognizing the advantages of the UAb teaching model for sustainability, from the perspective of its social dimension, namely that

it facilitates education for all and at all stages. This is evidenced as an innovation according to the STARS criteria [73]. Another strong point that the assessment highlighted is the university bet in sustainability teaching not only at formal programs from 1st, 2nd and 3rd level (bachelor to Ph.D.) but also at non-formal programs like open and massive on-line courses (see Table 4).

Through the assessment of the sustainability in UAb using STARS the key-actors were able to discuss ways of improvement namely new paths to implement sustainability practices (see Table 4). The improvements found are feasible and inexpensive but UAb should integrate sustainability into organizational practices and allocate financial and human resources in the next strategic plan. In particular, the application of STARS can be an important basis for the definition of a currently non-existent University sustainability strategy/policy, one of the fundamental pillars for the whole-school approach, which agrees with other authors [13,32]. In addition, the participatory process that took place during the completion and evaluation of the tool alerted and sensitized the focus group participants on sustainability issues at UAb, contributing to the fact that some of the proposals are already being implemented at the moment. Examples are the registration of the SDG in the resources uploaded in the Open Repository of the University, the development of more moments of joy and sociability (e.g., Christmas lunch) and ongoing process of the dematerialization of the administrative process.

**Figure 1.** Results of the Sustainability Tracking tool, Assessment & Rating System (STARS) in the six dimensions for Universidade Aberta (UAb) in the period of 2015–2017. Maximum scores calculated based on version of STARS 2.1 [58].

**Table 4.** Summary of the results of Assessment and Improvements for Universidade Aberta regarding the STARS dimensions.



• Decentralized Local Learning Centers in areas of low population density/close contact with society • Separate bins in all facilities • Engage all university community in an online collaborative platform for sustainable ideas

This tool was first used in Portugal and at a distance learning university. Its application in the UAb has also identified some points of better adaptation to European reality (in particular related to units of measurement and benchmarks) and distance learning universities (since these institutions do not have a formal campus with students). These adaptations were communicated to the tool implementation support services of STARS. Earlier studies indicate that e-learning has a lower impact on greenhouse gas emissions and climate change, as observed in the UAb assessment and according to other studies (e.g., [81]) but its direct and indirect impacts on sustainability need to be better studied, as also advocated by Findler et al. [14]. Indeed, the long-term impact of the practices being implemented and the impact of UAb on a more sustainable society are issues that the tool has not been able to assess on its own.

The scores result of the GM assessment from 2014 to 2018 at Universidad Autónoma de Madrid are available in Figure 2 and Table 5. The results showed that UAM is in a good position compared with other worldwide universities, namely a 55th position in a total number of 719 in 2018. In 2018 UAM reached the highest scores due to considerable improvement in the Education and Infrastructures dimension but in 2014 its performance was better in terms of Water, Waste, Energy and Transportation. These results do not exactly mean the worst performance in the following years since the number of indicators related to these dimensions changed in the GM tool. An overall 34th position in the ranking of 2014 was obtained in a total of 361 universities ranked (corresponding to half of the universities compared to 2018).

According to the Vice-chancellors focus group (2nd focus group—Table 2), one critical area where the university should make an extra effort is the outreach, highlighting that university is a place to develop sciences, knowledge and new solutions able to contribute and affect to the society, local and global community. This factor is not considered in this tool, although considered in STARS. Participants from this 2nd focus group also highlighted the importance of undertaking the SDGs agenda in the university activities towards sustainability.

For the student focus group, a key factor to be included in the future is "the communication strategy" to foster the environmental message in the university community. This item is partly included in UI Green Metrics, in the Education and Research criteria, which requires the existence of a university-run sustainability website and the existence of a published sustainability report (Green Metrics [50]). This improvement was also highlighted in the UAb case study where it was proposed for better engagement of all university communities the development of an online collaborative platform for sustainable ideas (see Table 4).

**Figure 2.** Scores of GreenMentrics for Universidad Autónoma de Madrid from 2014 to 2018 in total and in each dimension and also international ranking compare to total of other universities. Maximum scores calculated based on version of Green Metrics 2018 [50].

Common barriers for sustainability implementation in both Southern European institutions are related to low levels of community participation and sustainability awareness, financial constraints and a lack of HEIs legislative framework. Those barriers are also found in earlier studies (e.g., [76]). Some weaknesses found in UAb are also found in UAM, namely in terms of SDG disclosure and low sustainability awareness from the university community and lack of financial support. Of course, in terms of campus operations, both universities' reality is quite different due to their different sizes and regime of teaching. Nevertheless, they can learn from each other. For example, UAM, in terms of the Academic dimension can use teaching methods more based on new technologies like the ones used in distance learning at UAb, allowing also more transdisciplinary research and less theory (in accordance with authors like Lozano et al. [15]). UAM can also develop more lifelong learning programs or open courses to increase society sustainability awareness, like MOOCs, learning from the experience of UAb. In addition, UAb can learn from the experiences in terms of the SDG LAB and Eco campus initiatives that UAM uses and improve their worst performance in terms of Campus Operation. Even that UAb is a distance learning university without face to face students at the campus it has facilities where energy, water and waste measures can be implemented, learning from UAM experiences. More recently, at both Universities questionnaires are being developed to the students to understand their sustainability perceptions, engagement and motivations. With the questionnaires results new strategies can be put in place.

Looking into Figure 2 it can be seen that GM dimensions are mainly focused on environmental initiatives and actions on campus, hence, there are scarce indicators of policies, management, diversity, equity and community participation, what is in accordance with several authors (e.g., [17,29]). Also, this ranking's main weakness could be considered that the information provided by the universities many times do not include evidence or in-situ verification. According to the experiences of the Eco campus focus group, the information sometimes can be considered subjective according to the understanding of the person in charge to provide the data. In the case of STARS, the data is validated by the rector team and also externally certified (like in UAb case study). However, the main strength is the accessibility and comparability of the outcome among the years (also in accordance with other authors like Lauder et al., [49]). The results from all university participants are available in the Green Metrics, easily accessible to compare by regions, countries and universities. From the benchmarking point of view, it could also facilitate the decision-making process for managers based on the potential analysis of the progress on these factors. The level of understanding is high and the authors of the GreenMetrics have provided the criteria, indicators and methodology used on their website (see for example the 2018 guide [50]).

**Table 5.** Summary of the results of Assessment and Improvements for Universidad Autónoma de Madrid regarding the STARS dimensions.


**Table 5.** *Cont.*


#### • ECOCAMPUS is a leader of environmental programs at HEIs since 1997

#### *5.3. Overall Discussion*

The tools critical analysis on the tools and cases studies application, still raise the question of the effect of implementing these tools versus the actual integration of sustainability into an HEI. The integration of sustainability in HEIs should come along with the modification of existing structures and habits, which creates many challenges related to the involved actors, the available resources, values and strategic choices to be made [82,83]. Also, based on Kapitulcinova et al. [9], the "transformational change" should occur at the level of the entire HE. It is therefore essential that a critical mass of units comprising the institution adopt sustainable development principles in their respective tasks and duties. In addition, according to Alonso-Almeida et al. [84] and Beringer et al. [85], to achieve the sustainability maturation, the sustainability integration at HEIs should involve all dimensions into a whole-school approach.

For all those reasons it is considered that the implementation of integrative approaches and models still needs to be encouraged and further research is needed [9,14,31,76]. Regardless of this fact, the case studies demonstrated that the implementation of the tools make possible to assess the state of implementation of sustainability in HEIs, monitor it, communicate it, share it within and outside the organization and improve and stimulate change, often enabling low-cost measures to be implemented. These statements are also advocated by other authors (e.g., [16,24,28,32]).

A change process enforces an overall vision, an increasing need for change that is experienced by the stakeholders, resources to support the process and short-term gains that can be communicated [86]. In this research the application of the assessment tools was complemented with participatory processes. While participatory methods are not commonly considered in these tools, they are central support for more holistic implementation allowing for the best man-nature link and a reflection that can better respond to the transformation of the institution and individual towards sustainability [66]. These tools must have a stronger component of student community participation and involvement, as this community is a major agent of change. Given the characteristics of experimentation and research of HEIs, indicators that can be incorporated into these tools or that can be used independently, should be tested to allow a long-term evaluation of whether the transformation process has been successfully achieved. Disterheft et al. [66] suggested examples of these types of indicators, based on perspectives such as the whole-school approach, interconnection between man and nature, community cohesion, celebration and happiness and principles of democracy. As argued by Ramos [65] the challenges in the area of sustainability assessment indicators should be based on transdisciplinary, collaborative and innovative scientific development where communities and the individual play a central role. Greater emphasis should be given to the development of indicators to assess non-traditional aspects of sustainability, such as ethics, culture and art, aesthetics, governance efficiency, spirituality, solidarity, compassion and trust, which represent fewer tangible dimensions of society [65,87].

These tools are also too operational not evaluating the strategic processes, as also stressed by Arroyo [33]. being able to incorporate the unpredictable and not only knowing how to deal with linear problems but also being able to assess what external impact HEIs have in practice on sustainability and going beyond the limits of HEIs [14]. As an example, the impact assessment of the research that is developed in HEIs on ESD should be carried out. Since this cannot be done based on a simple citation counting and bibliometric analysis because they do not allow to accurately define the result of this same investigation in the SD, more qualitative and documentary analyses are necessary for a more robust evaluation [14].

Few studies have explored and evaluated the role of HEIs as agents of change [31]. Also, there is a weak link between HEIs and external networks and key actors and with local and regional policies, what does not contribute to change in organizational management [82], holistic integration of sustainability in HEIs and their impact abroad [14,88]. The external impact of implementing sustainable development in HEIs can be measured in the local economy and culture, in challenges in society, in the natural environment and in the policies. Impacts can be short-term and direct, such as by training skilled workers or reducing greenhouse gases in the facilities but also indirect and long-term, such as changing graduates' sustainable lifestyles or in the implementation of Sustainable Development Goals (based on [14] suggestions).

#### **6. Conclusions**

A high increase has been seen in the implementation of sustainable development principles into Higher Education Institutions and in the research to assess its performance. In this article, a critical analysis of the existing tools to assess and benchmark ESD implementation was conducted by four judges according to a list of criteria. There are limitations associated with these kinds of critical review and qualitative evaluation but those limitations were weighed up in the mains findings. The tools applicability was then discussed in two case studies. The tool STARS—Sustainability Tracking, Assessment & Rating System was the one that was better classified in terms of understanding, comparability, sustainability broadness and support to decision, being one of the tools more worldwide used by different HEIs. The tool STARs was also used in one of the case studies. So, the use of STARS could be a good choice for HEIs that would like to assess and benchmark their sustainability performance according to a holistic and integrated approach.

The tools currently available for evaluating sustainability initiatives in HEIs do not all have the same objective or do not homogeneously evaluate the implementation of sustainability. As a consequence, some ambiguity is translated to its actual implementation and real contribution to the transformation for change. It is therefore recommended to set common general sustainability objectives in HEIs considering the integrative whole-school approach, regardless of some regions of institutions specificities (which may be indicated by specific objectives). Some improvements to the tools were discussed in this research and suggestions for future research. Sustainability assessment in HEIs should be viewed as a social construction, emerging from the different partners involved and according to mixed, bottom-up, top-down approaches, where the various actors, internal and external, contribute positively to the implementation of the sustainability assessment tool.

The implementation of Sustainability assessment tools in the HEIs case studies has shown that its use is an important driving force for the first diagnosis, a source for defining ways of improvement and also for future changes in organizational management. Also, the application of the tools in two HEIs of neighbor countries complemented with participation activities with key-actors, allowed to show the weakness of these tools namely the lack of the assessment of the impact of the sustainability performance of the university in the society and their real contribution for a sustainability transition. Bearing in mind that both countries have some similarities, namely cultural, this collaboration also brought up difficulties that need to be addressed in order to apply and implement EDS successfully. Barriers were identified and examples of improvement were given. Nevertheless, both universities learned from each other besides their differences and future developments for their sustainable performance improvement are already in place.

This article adds new insights about the main characteristic, common designations and objectives this type of tools should have and defined a list of criteria for their evaluation, trying to uniformize their meaning. Also, this research gives suggestions about improvements that are still needed on these tools so they can more fully answer their main purpose. Improvements can be in terms of incorporating in the tools an integrated process of stakeholder's participation, adding of indicators to assess non-traditional aspects of sustainability and being able to assess what external impact HEIs have in practice on sustainability.

**Author Contributions:** S.C. led the process of the tools review and assessment and overall discussion of the results. R.M. and S.C. developed the case study in UAb and wrote the respective methods and results. L.A.S.H. and C.E.B.A. developed the literature review and case study in UAM and wrote the respective methods and results. All the authors classified the tools according to the listed criteria and contribute for the article writing, revision and approval of the manuscript. All authors have read and agreed to the published version of the manuscript.

**Funding:** CENSE is financed by Fundação para a Ciência e Tecnologia, I.P., Portugal (UID/AMB/04085/2019). The funding sources played no part in the design, analysis, interpretation or writing-up of the paper or in the decision to publish.

**Acknowledgments:** The authors wish to thank all participants in this research in the different focus groups and workshops held in UAb and UAM, in particular Ana Paula Martinho and Teresa Teixeira in the data collection at UAb. The authors want also to thanks the rectorate support on the assessment processes in each university, in particular to Vice-cancellor for Qualaty Carla Oliveira in UAb and Santiago Atrio Cerezo Vice-cancellor for Sustainability and Campus and Jesús Rodríguez Pomeda, Vice-cancellor of Strategy and Planning in UAM.

**Conflicts of Interest:** The authors declare no conflicts of interest.

#### **References**


© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel. +41 61 683 77 34 Fax +41 61 302 89 18 www.mdpi.com

*Sustainability* Editorial Office E-mail: sustainability@mdpi.com www.mdpi.com/journal/sustainability

MDPI St. Alban-Anlage 66 4052 Basel Switzerland

Tel: +41 61 683 77 34 Fax: +41 61 302 89 18