Using Modified Delphi Study to Develop Instrument for ESG Implementation: A Case Study at an Indonesian Higher Education Institution

Abstract

Most research states that implementing environmental, social, and governance (ESG) has positive impacts. However, fewer studies have discussed ESG implementation in higher education. This study aimed to develop instruments to assess the ESG atmosphere in higher education institutions. A modified Delphi approach was employed. Experts were invited from a private higher education institution in Indonesia. A deductive study, discussion, and two stages of getting consensus from panelists were conducted. The instrument was distinguished into four types for four groups of higher education stakeholders: Students, Staff, Faculty Members, and Community Members. The I-CVIs ranged from 0.80–1.00, while the minimum values of S-CVI/Ave and S-CVI/UA were 0.98 and 0.91, respectively, meaning the content validity was excellent. The final version instrument has been tested and declared valid, reliable, and ready to be used for empirical research for universities to assess their contribution to the Sustainability Development Goals (SDGs). There are also opportunities to conduct further research on the existence of recursive and non-recursive models between factors.

1. Introduction

In 2015, all United Nations (UN) members adopted the 2030 Agenda for Sustainable Development Goals (SDGs). This agenda provides 17 Sustainable Development Goals (SDGs): (1) No poverty; (2) Zero hunger; (3) Good health and well-being; (4) Quality education; (5) Gender equality; (6) Clean water and sanitation; (7) Affordable and clean energy; (8) Decent work and economic growth; (9) Industry, innovation and infrastructure; (10) Reduce inequalities; (11) Sustainable cities and communities; (12) Responsible consumption and production; (13) Climate action; (14) Life below water; (15) Life on land; (16) Peace, justice and strong institutions; and (17) Partnerships for the goals [1]. Organizations or companies around the world are expected to participate in the achievement of some or all of these goals based on their respective abilities. Previously, companies, especially financial firms, have successfully implemented environmental, social, and governance (ESG) principles to support the company’s achievement and achieve SDGs simultaneously. ESG was first mentioned by the UN in 2006 when it launched the Principles for Responsible Investment (PRI) [2]. PRI suggested that investors should pay attention to other concerns rather than just focusing on profit.

Most research states that implementing ESG in financial firms has positive impacts: overall and individual ESG performance were significant positive indicators of creditworthiness as measured by credit rating [3], ESG risks have the potential to reduce financial returns, and most superannuation fund members are interested in ESG investing [4], conventional funds consider the firms in which Socially Responsible Investment (SRI) funds invest to integrate ESG criteria [5].

Higher education has been identified as the crucial driver in achieving the SDGs, believed to influence people’s thoughts and actions, and plays an essential role in creating public awareness regarding the consequences and impact of unsustainability action on society [6,7]. Many higher education institutions have achieved the SDGs using their respective strategies. Universities conduct many activities related to sustainability, including environmental issues [8,9,10,11,12,13,14], social issues [9,15,16,17,18,19], and governance issues [14,20]. Universities can imitate the successful application of ESG principles from financial companies to contribute towards achieving the SDGs. However, fewer studies have discussed ESG implementation in higher education. One reference states that the higher education sector is early in its ESG reporting journey [21]. Another reference argues that universities must carefully plan strategies to promote a broader ESG strategy, making higher education institutions attractive to stakeholders [22]. The implementation of ESG in higher education institutions will simultaneously improve their quality. This is supported by the fact that the global higher education institution ranking conducted by Quacquarelli Symonds (QS Rank) has included the ESG indicators in the requirements [23].

Higher education in Indonesia has also begun to participate in sustainability. The University of Indonesia GreenMetric (UIGM) World University Ranking was launched in 2010. This ranking captures the latest conditions and policies related to Green Campus and Sustainability in universities worldwide [24]. The report stated that 101 higher education institutions in Indonesia participated in the UIGM, even though Indonesia has 4546 higher education institutions. This condition shows that there is still a wide gap between the number of higher education institutions concerned with sustainability and the total number of higher education institutions. Higher education in Indonesia still needs encouragement to participate in achieving the SDGs. One form of encouragement is providing references to them. A previous study has outlined the steps to becoming a sustainable university and determined the parties involved [25]. This study intended to expand knowledge of education by outlining the factors and indicators that support the implementation of ESG to achieve the SDGs. The available indicators were derived into items and formed into ready-to-use instruments. Thus, this study contributes to further research aiming to conduct empirical studies that prove the relationship between factors using this ready-to-use instrument.

2. Theory and Context of ESG

For decades, people have believed that modern businesses are responsible to their stakeholders, communities, and society. Stakeholders are increasingly looking for non-financial elements to help them make the best business decisions [26]. As a result, the organization’s information must be used to reveal valuable and trustworthy information and monitor corporate officials. ESG has become a competitive component for modern organizations; therefore, ESG’s responsibility to all stakeholders appears to be crucial [27]. This ESG implementation will be carried out by involving or impacting stakeholders both internally and externally.

Higher education institutions worldwide are being challenged to become more focused, efficient, and successful to meet the needs of a globalized and technologically advanced higher education market [28]. Universities adopt a business enterprise mindset to ensure survival in the face of resource constraints and rapid change in their working environment. Implementing ESG in universities is an opportunity to develop and improve the organization’s quality and simultaneously contribute to achieving the SDGs.

The global disruption caused by COVID-19 has had several environmental and climate consequences. Due to movement restrictions, air quality in many cities has improved, and water pollution has decreased. Furthermore, the increased use of personal protective equipment (PPE, e.g., face masks, hand gloves, etc.), their haphazard disposal, and the generation of a large amount of hospital waste have negative environmental consequences [29]. The global COVID-19 pandemic has also forced the temporary physical closure of schools and higher education institutions. Approximately 220 million students worldwide have been affected in higher education, presenting policymakers and educational institutions with unprecedented challenges [30]. Higher education must ensure that underrepresented, vulnerable, and disadvantaged learners are not left behind. The pandemic also emphasizes the value of university–community engagement. The pandemic has also directly impacted how universities operate and university governance, with management staff required to make various emergency decisions while allowing for additional flexibility. Those facts show that in the post-COVID-19 period, higher education needs to pay attention to environmental, social, and governance concerns.

3. Materials and Methods

The Delphi method has been proven as one of many techniques for organizing a group communication process so that a group of people can efficiently deal with a complex problem [31]. However, Gooman (1987) in Mao [32] said that a traditional Delphi method begins with an open-ended questionnaire, is time-consuming, and usually leads to a low response rate. Therefore, a study that successfully developed a tool for measuring healthcare has designed a modified Delphi method that used a structured questionnaire to ask for experts’ opinions [32]. The structured questionnaire can be developed based on extensive literature reviews or a focus group interview study. This study adopted the modified Delphi method to compile the instrument. Furthermore, piloting was conducted to test the validity and reliability of the instrument to ensure that the instrument was ready to use. The research framework is shown in Figure 1, containing four steps in the modified Delphi method and one piloting step.

3.1. Stage 1: Determine Factors and Indicators

Three academics were involved in the discussion to determine what factors were related to the implementation of ESG. This discussion found that the three factors, namely Leadership, Strategy, and Performance, were closely related to the implementation of ESG. The discussion was combined with a deductive approach to corroborate the results. The following literature review has affirmed those findings.

The Sustainability Accounting Standards Board (SASB), a credible independent nonprofit organization, identifies Leadership as the factor affecting the organization’s performance on ESG issues [33]. Leadership involves managing issues inherent to the business model and common practice in the organization, an essential feature in the initiation step of effective governance and important to becoming a successful world-class university [34,35]. Any organization needs strategies to achieve performance. Likewise, in the field of education, strategies are required by higher education to deal with complicated problems and the competitive knowledge marketplace that has appeared in the last decade [36,37]. A new perspective requires that the performance of nonprofit organizations is also determined by how well and efficiently they meet the needs of their stakeholders [38]. Universities have unique, measurable performance related to stakeholders, for example, satisfaction in education services received (teaching–learning service, research service, community service, administrative service, and curricular service) [39].

The proposed factors were latent variables that required measurable indicators. Therefore, the discussion continued with determining indicators. The ideas were collected from previous research. Similar arguments were grouped and given a name and identification code. Twenty-eight indicators have been found as selected indicators (

Table 1. List of proposed indicators.

Factor	Reference		Proposed Indicator	Code
	Statement	Source
Leadership	Leadership ability is essential. Leaders can influence changes in the organization.	[40]	Rector/leader’s leadership	IL1
	Universities are looking for good leaders who can advise and implement change, foster collaboration and teamwork, and act as coaches when needed	[41]
	Leadership encourages sustainability embedding into all university areas	[14]
	Vision and mission lead to becoming a sustainable university	[42]	Vision and mission	IL2
	Leaders use their power by using resources to affect organizational change.	[40]	Resource input	IL3
	The educational resources in higher education include student intake, qualified staff, facilities and equipment, staff–students ratio, and financial support.	[43]
	Sustainability advocates, education, and awareness are important.	[14]	ESG Education	IL4
	Cultural differences affect leadership performance. It is recommended to use communication to build consensus within the group.	[40]	Culture and consensus	IL5
Strategies	Strategy is essential to improve a firm’s ability through the execution of the mission.	[44]	Strategi for ESG	ISt1
	It is essential to create a good plan design of sustainability in higher education to avoid gaps between plan and implementation.	[45]	Plan of Strategies	ISt2
	Higher education must have a strategy to be efficient and consider stakeholders’ expectations and demands.	[10]
	Funding is crucial to becoming a successful world-class university.	[35]	Funding planning	ISt3
	Sustainability programs could become a source of money for additional upgrades.	[8]
	How well higher education manages funding is one measure of performance.	[46]
	Government funding was an essential driver for higher education to achieve SDGs.	[47]
	The execution of sustainability programs is an important thing to be concerned about by the university.	[48]	Execution	ISt4
Environmental	Students must be involved in water management projects on campus.	[10]	Energy-saving (electricity and water)	IE1
	Electricity efficiency is crucial in higher education. It can increase cost-effectiveness and reduce greenhouse gas emissions. Modifying, upgrading, and maintaining electrical facilities can improve energy efficiency.	[8]
	Universities can invest in renewable energy. Universities need to reduce environmental impact in the water area.	[14]
	Universities need to be sensitive about energy conservation by placing signs of measure and warning.	[9]
	Universities can emphasize student waste reduction and recycling.	[10]	Waste reduction and recycling	IE2
	Changes in building and infrastructure to reduce energy consumption.	[11]	Green building and campus	IE3
	Universities need to have adequate greenfield.	[9]
	Buildings must respond to climate change because buildings often have a long lifespan. Making effective selection concerning building attributes is critical to achieving sustainability goals.	[12,13,49]
	Universities need to be concerned about a sustainable and efficient transport systems and environment friendly transport such as electric vehicles.	[10]	Transportation (bicycle, electric buses, electric cars, etc.)	IE4
	Universities need to reduce environmental impacts in the transport area.	[14]
Social	All university members should respect human rights, cultural diversity, intercultural understanding, peace and human security, and gender equality.	[15]	People and respect	ISo1
	University Social Responsibility (USR) contributes to the effective transformation of society by resolving exclusion, inequality, and sustainability problems.	Vallaeys (2007) in [17]
	GRI 405 highlights equal opportunities for employees regardless of gender or other relevant diversity such as minority or vulnerable groups. Concerning universities, this can also be related to equality for students.	[16]	Gender and racial equality	ISo2
	The meaning of minority can be broadened as racial differences.	[9]
	Education in human and social values fosters civic solidarity, respect for diversity, and equal opportunities among students.	[18]	Social values education	ISo3
	Social values indicate changes in the lives of individuals or groups using tangible and intangible resources by social actors, which creates social change in society.	[19]
	The meaning of vulnerable groups can be defined as disabled students or students who have economic deficiencies.	[9]	Care for weak students	ISo4
	Social relationships with firms, local communities, NGOs, and other organizations are essential.	[15]	Social relationships	ISo5
Governance	Corporate governance refers to the standardized interaction among the many players, including shareholders, management, boards of directors, employees, customers, financial institutions, regulators, and the community involved in directing and controlling private firms.	[50]	University autonomy	IG1
	Governance performance features authority, human resource development and empowerment, and information management capacity. The feature authority means that institutions have the right to make regulations, organizational mandate issues, roles, and responsibilities among management levels identified.	[34]
	Authority in the program, curriculum, and human resources is essential in higher education governance. This authority refers to academic freedom.	[20]	Academic freedom	IG2
	The faculty members have supported a clear policy to integrate sustainability into the curriculum and research in the academic area.	[14]	Professor authority	IG3
	Authority in the program and curriculum and human resources also refers to the authority of professors.	[20]
	The ability of staff to plan, implement, monitor, and evaluate activities. Thus, internal stakeholders can influence and control the implementation of activities.	[34]	Staff empowerment	IG4
	Making sustainability initiatives visible in public is essential to sustainability governance in HEIs. Stakeholder opportunities to observe issues, positions, activities, and results.	[51]	Information transparency	IG5
Performance	The framework contains how well higher education manages its cost, how well higher education contributes to human capital formation, how well higher education contributes to innovation, and how well higher education contributes to social, cultural, and environmental development.	[46]	ESG evaluation	IP1
	Targeting and monitoring have been widely recognized for evaluating sustainability performance.	[45]
	Success of nonprofits should be measured by how effectively and efficiently they meet the needs of their stakeholders.	[38]	− Student satisfaction − Staff satisfaction − Professor satisfaction − Social satisfaction	IP2 IP3 IP4 IP5
	Universities engage with each stakeholder according to their roles.	[14]
	Within the scope of the university, the stakeholders involved in building a sustainable campus include internal stakeholders, namely students, faculty members/professors, staff, and external stakeholders, namely alumni, community, company, government (in this case referred to as social relations).	[52]

). Furthermore, confirmation and consensus from the panelists were required before moving on to the following process.

3.2. Stage 2: Verify Factors and Indicators

In the Delphi method, the consensus of expert panelists must be obtained [53]. The consensus was obtained by analyzing the responses of the panelists. Nine experts have been invited to participate as panelists. Crawford & Wright [54] recommend obtaining the opinion of 5–20 experts with appropriate domain knowledge and heterogeneous background. A total of eight experts stated their willingness to be involved in this study. They were leaders from a private university in Indonesia with different backgrounds and position levels.

Table 2. Expert demographics.

Number of Experts Invited	:	9
Number of experts who expressed willingness	:	8
Position
University level	:	Rector (1) Vice-rector (1)
Faculty level	:	Dean (2) Vice-dean (1)
Program level	:	Program chairperson (3)
Background	:	Engineering Business Social sciences Natural sciences Psychology

shows the demographics of the experts.

Panelists were asked whether the proposed indicators were appropriate to support the relevant factors. Experts responded by giving a score of 1 to 5.

Table 3. Suitable scale.

Score	Scale Reference	Definition
1	Most unsuitable	No relevance. It should be dropped as an item to consider.
2	Unsuitable	Insignificantly relevant to the latent variable. It has a bit of impact.
3	Moderately suitable	Maybe relevant to the latent variable.
4	Suitable	Relevant to the latent variable.
5	Very suitable	It is most relevant to the latent variable. It has a direct bearing on the latent variable.

Source: adapted from Jillson [55].

explains the definition of the suitable scale.

In this study, the statistical analysis was conducted by following the steps of the previous research that has successfully developed an instrument [32]. The data from panelists’ opinions were inputted into SPSS (Statistical Package for the Social Scientists) software version 22.0 for Windows (IBM, Armonk, NY, USA). To obtain a consensus, measurement with a Likert scale of 1–5, the median, interquartile range (IQR), and the level of the agreement would be evaluated. The median should be between the top two measures (suitable/very suitable), the IQR should be one or less, and for the level of the agreement, the answers from the top two measures should be more than 70%.

3.3. Stage 3: Determine Items and Stakeholders

The process was continued by compiling items derived from each indicator. A total of 78 items were composed (

Table 4. List of items.

Indicator Code	Item Code	No	Item
IL1	Le1	1	University leaders have concerns about ESG implementation.
	Le2	2	The leader gives directions for ESG implementation.
	Le3	3	The leader encourages ESG in all university areas.
IL2	Le4	4	The university’s vision and mission have considered ESG.
	Le5	5	The university’s mission provides a clear understanding of becoming a sustainable university.
IL3	Le6	6	Leaders direct the use of resources (human, facilities, finance, etc.) to support ESG implementation.
	Le7	7	University policies related to the use of resources (human, facilities, funding, etc.) always consider ESG issues.
IL4	Le8	8	The university provides awareness about understanding ESG through meetings, training, or other media.
	Le9	9	I have participated in ESG awareness activities at least once.
IL5	Le10	10	University leaders encourage the ESG culture to inspire university members in carrying out daily activities.
	Le11	11	I agree with the ESG implementation in the university.
	Le12	12	ESG has inspired students, employees, and community members in their daily activities.
ISt1	St1	13	The university has established a master plan to achieve the objectives.
	St2	14	The university leaders cascade the master plan/long-term strategic plan to the annual plan/operational plan.
	St3	15	University leaders establish the master plans by considering efficiency and effectiveness.
ISt2	St4	16	The university has a master plan and operational plan regarding the implementation of ESG.
	St5	17	The university provides operational guidelines to facilitate ESG implementation.
ISt3	St6	18	The university allocates funds for activities related to ESG implementation.
	St7	19	ESG programs can be financially beneficial for universities (e.g., through efficiencies of electricity and water).
	St8	20	Funding from the government is important for universities to implement ESG.
ISt4	St9	21	University leaders have the competencies to implement ESG.
	St10	22	The university evaluates and monitors the implementation of ESG.
	St11	23	The university makes improvements to previous deficiencies or failures.
IE1	En1	24	The university promotes energy and water efficiency (e.g., put on signs, stickers, labels about energy-saving and water-saving).
	En2	25	The university uses energy-saving lights.
	En3	26	The university maintains or upgrades electronic equipment to support energy savings.
	En4	27	The university utilizes renewable energy sources (e.g., solar panels and wind power).
	En5	28	The university encourages research or ideas for the creation of renewable energy.
	En6	29	The university maintains the environment around the water area (e.g., ponds, lakes, rivers, seas).
	En7	30	The university engages its stakeholders in implementing energy and water-saving projects.
IE2	En8	31	The university provides trash bins with organic and non-organic waste separation in the campus area.
	En9	32	The university encourages respondents to reduce the use of materials that cause waste (e.g., reduce the use of paper and plastic).
	En10	33	The university sets a target regarding the reduction of using materials that cause waste (e.g., targets to reduce the use of paper and plastic).
	En11	34	The university processes organic waste into something useful.
IE3	En12	35	When constructing a new building or renovating an established one, the university considers energy savings (e.g., during the day, the room does not need lights, and windows are available to minimize air conditioning use).
	En13	36	The university implements a green roof approach building (a building’s roof that is partially or entirely covered with vegetation and a growing medium).
	En14	37	The university has adequate greenfield areas.
IE4	En15	38	The university promotes the use of environmentally-friendly transportation (e.g., bicycles and electric vehicles).
	En16	39	The university provides adequate facilities for environmentally-friendly transportation users (e.g., special parking for bicycles and electric vehicles).
	En17	40	The university plans to reduce fossil-fuelled official vehicles to environmentally friendly ones (e.g., electric vehicles).
ISo1	So1	41	The university members respect human rights and strive to maintain peace and security.
ISo2	So2	42	University members respect gender equality.
	So3	43	The university members respect racial differences.
	So4	44	The university members give equal treatment to minorities and majorities (all forms of minority and majority).
ISo3	So5	45	The university provides education on human and social values to foster civic solidarity, respect for diversity, and equal opportunities among students, staff, and community members.
	So6	46	The university encourages individuals or groups to use available resources to become actors who create social change in society.
ISo4	So7	47	The university is concerned with the needs of students with disabilities.
	So8	48	The university is concerned with the needs of students who have economic deficiencies.
ISo5	So9	49	The university maintains relationships with the local community around campus.
	So10	50	The university maintains relationships with community members where service activities are carried out.
	So11	51	The university maintains relationships with NGOs or other organizations related to social activities.
IG1	Go1	52	The university leaders have the authority to define the university’s vision and mission.
	Go2	53	The university leaders have the authority to determine the university’s grand design and operational plan.
	Go3	54	University leaders have the authority to provide guidance and make regulations.
	Go4	55	University leaders have to be responsible for the university’s performance.
IG2	Go5	56	The study program has the freedom to determine the curriculum.
	Go6	57	The study program has the freedom to determine the research and community service roadmap.
IG3	Go7	58	In implementing the curriculum, lecturers have the flexibility to determine lecture materials and learning models.
	Go8	59	In implementing the research and community service roadmap, lecturers have the flexibility to determine their research and service activities.
	Go9	60	Lecturers have the freedom to deliver or publish ideas or opinions based on academic thoughts or research results.
IG4	Go10	61	Administrative and supporting staff are involved in preparing/planning university programs.
	Go11	62	Administrative and supporting staff are involved in implementing university programs.
	Go12	63	Administrative and supporting staff are involved in evaluating university programs.
IG5	Go13	64	University leaders disseminate university programs to all stakeholders through various means such as meetings, flyers, websites, social media, etc.
	Go14	65	University leaders disseminate university performance to all stakeholders through various means such as meetings, flyers, websites, social media, etc.
	Go15	66	The university provides opportunities for stakeholders to give input or opinions on university activities and performance.
IP1	Pe1	67	The university evaluates performance related to environmental concerns.
	Pe2	68	The university evaluates performance related to social concerns.
	Pe3	69	The university evaluates performance related to governance concerns.
IP2	Pe4	70	I am involved in university activities related to environmental conservation (saving electricity and water, reducing waste and recycling, and environmentally-friendly transportation).
	Pe5	71	I am involved in community service activities organized by the university.
	Pe6	72	I get equal treatment in non-academic activities.
	Pe7	73	I get equal treatment in academic activities.
	Pe8	74	I get enough information about the university’s programs or activities related to ESG.
	Pe9	75	I get enough information about the university’s performance regarding ESG.
	Pe10	76	The curriculum in my study program contains material on ESG/sustainability/corporate social responsibility or other similar terms.
	Pe11	77	I have the freedom to include ESG content in my courses.
	Pe12	78	I have the freedom to research the ESG theme.

). At the same time, the academics found the stakeholder groups mentioned most by previous studies. The stakeholder groups were Students (Stu), Staff (Sta), Faculty Members (FM), and Community Members (CM) [52,56,57,58,59,60]. Furthermore, a consensus from the panelists was needed to verify the suitability of the items to be asked to stakeholders.

3.4. Stage 4: Verify Items

In this stage, panelists were asked to rate the suitability of an item for a particular stakeholder group. Answers were on a scale of Yes/No. The answer YES indicated that the panelists agreed to address an item to the designated stakeholders, while the answer NO represented the converse.

The consensus was achieved when more than 67% of the panelists gave the same answer [61]. In this case, if more than 67% of the panelists answered YES, it is considered that the item was suitable for a particular stakeholder group. Items that agreed YES were retained, otherwise they were eliminated. The remaining items would become the prototype instrument aimed at specific stakeholder groups.

An analysis of the content validity index (CVI) was conducted. Content validity is an important procedure in scale development and is the degree to which an instrument has an appropriate sample of items for the measured construct [62]. Content validity was computed for each item (I-CVI) and overall scale (S-CVI), including the average (S-CVI/Ave) and the universal agreement (S-CVI/UA). I-CVI was computed as the number of experts giving YES for each item divided by the total number of experts. S-CVI/Ave was calculated by dividing the sum of the I-CVIs by the total number of items. Universal agreement (UA) was obtained by assigning a value of 1 to the item where all panelists answered YES; otherwise, they were given a value of 0. Then, S-CVI/UA was calculated by dividing the sum of UAs by the total number of items [63]. The statistical significance values for I-CVI, S-CVI/Ave, and S-CVI/UA were set at >0.78, 0.9, and 0.8, respectively [64].

3.5. Stage 5: Piloting the Instrument Prototype

Based on the verified prototype instrument for each stakeholder group, the instruments were distributed to stakeholders. The stakeholders were from the same university as the panel experts. This test was intended to reveal the instrument’s reliability and validity. All tests were carried out using SPSS 22 software with α = 0.05.

Table 5. The decision-making rules regarding validity and reliability.

Test	Decision	Purpose
Validity	Pearson’s correlation > Critical value Pearson Product Moment or p-value < 0.05 The critical value depends on the number of samples (N), as listed in the R Product Moment Table	To confirm that the item can represent the indicator
Reliability	Cronbach’s Alpha > 0.60	To confirm the consistency of the instrument

shows the decision-making rules regarding validity and reliability.

4. Results

Stage 1 found that leadership success can be measured by indicators of how a leader influences all members regarding ESG implementation through the vision and mission and utilizing available resources, socializing the ESG to become a culture and consensus among members. Strategy is crucial for implementing the mission, including the implementation of ESG. It includes financial planning and execution stages. Strategy development is needed to avoid gaps between planning and implementation. The presence of students and staff on campus greatly contributes to the environment around the campus. Therefore, the following issues are relevant to environmental concerns in the campus area: energy-saving (electricity and water), waste reduction and recycling, green campuses, and the impact of transportation systems. University activities are closely related to social relations, both on internal and external campuses. Therefore, the issue of gender and racial equality, as well as people’s respect, is very close to campus life. In addition, universities should also pay attention to students who are disabled or have economic weaknesses and maintain good relations with the community. Campus governance is needed to facilitate operational activities. Therefore, universities need to have the authority to regulate it. Academic development is entrusted to the study program and lecturers to manage it. The staff has the right to contribute to and control operational activities. Likewise, stakeholders have the right to obtain accountable and transparent information. Meanwhile, to measure performance, it is necessary to evaluate the plan and implementation, as well as measure the level of satisfaction of stakeholders regarding the implementation of ESG. The measurable indicator descriptions in this study have been systematically presented in Table 1, becoming references for the items’ compilation.

The data from Stage 2 was processed with SPSS and showed the results in median, IQR, and level of agreement, as shown in

Table 6. The median, IQR, and level of agreement of items.

Factor		Indicator	Median	IQR	Level of Agreement
Leadership	IL1	Rector/leader’s leadership	5	1	1
	IL2	Vision and mission	5	1	1
	IL3	Resource input	4	0	0.875
	IL4	ESG Education	4.5	1	0.875
	IL5	Culture and consensus	5	1	1
Strategy	ISt1	Strategy for ESG	4.5	1	1
	ISt2	Plan of Strategies	4.5	1	1
	ISt3	Funding planning	4	1	0.75
	ISt4	Execution	5	1	1
Environmental	IE1	Energy-saving (electricity and water)	5	1	1
	IE2	Waste reduction and recycling	5	0	1
	IE3	Green building and campus	5	0	1
	IE4	Transportation (bicycle, electric buses, electric cars, etc.)	4	1	1
Social	ISo1	People respect	5	1	1
	ISo2	Gender and racial equality	5	1	1
	ISo3	Social values education	4.5	1	1
	ISo4	Care for weak students	5	1	1
	ISo5	Social relationship	4.5	1	1
Governance	IG1	University autonomy	5	1	0.875
	IG2	Academic freedom	4.5	1	1
	IG3	Professor authority	4	1	0.875
	IG4	Staff empowerment	4.5	1	1
	IG5	Information transparency	5	1	0.875
Performance	IP1	ESG evaluation	4	1	1
	IP2	Student satisfaction	5	1	1
	IP3	Staff satisfaction	4.5	1	0.875
	IP4	Professor satisfaction	5	1	1
	IP5	Social satisfaction	4.5	1	1

. The median values were in the range of 4 to 5, all IQRs were <1, and the levels of agreement were >0.70. These results indicated that all indicators were considered suitable to support the factors. The Funding planning indicator obtained the worst achievement with a median of 4, IQR of 1, and the lowest level of agreement (0.75). Even though the results were not good, they were still in the agreement range. These verified indicators were then derived to 78 items, as was done in Stage 3.

The Leadership Factor, which has five indicators, was derived into twelve items. The items compiled include questions about how a leader performs concerning directing, compiling a vision and mission about ESG, utilizing resources, encouraging members to participate, increasing awareness about ESG, and making ESG an inspiration and culture for the organization. The four indicators from the Strategies Factor were derived into eleven items, including the existence of a master plan and operational plan, especially those related to ESG, financial planning, leadership competence in implementing ESG, and evaluation and improvement of ESG implementation. Four indicators in the Environmental Factor were derived into seventeen items: how the university carries out electricity and water efficiency, encourages research and utilizes renewable energy, waste management, building and greenfield areas, and the provision of facilities and encouragement towards environmentally friendly transportation. Five indicators in social factors were derived into eleven items: respect for human rights, gender, and racial equality, solidarity, concern for disabled or economically weak students, care for the community, and other social organizations. In the Governance Factor, five factors were derived into fifteen items, including the university authority in preparing the vision and mission; planning; regulation; performance; dissemination of programs and achievements to stakeholders; study programs’ authority to manage curriculum and research; freedom for lecturers to apply lecture methods and materials, as well as carry out research and community service; and the involvement of supporting staff in preparing, implementing, and evaluating university programs related to ESG. Items in the performance factor involve assessing the implementation of ESG and stakeholders’ satisfaction in academic and non-academic activities and services, mainly those most commonly found with the application of ESG. There were twelve items arranged.

The items were arranged to be distributed to the respondents. Therefore, the questions must be easy to understand so that respondents will have a similar understanding of them. A total of 78 items were composed (Table 4).

Table 7. The level of agreement between the items.

Indicator Code	Item Code	Level of Agreement				Indicator Code	Item Code	Level of Agreement
		Stu	Sta	FM	CM			Stu	Sta	FM	CM
IL1	Le1	1.00	1.00	1.00	1.00	ISo1	So1	1.00	1.00	1.00	1.00
	Le2	0.60 *	1.00	1.00	0.20 *	ISo2	So2	1.00	1.00	1.00	1.00
	Le3	1.00	1.00	1.00	1.00		So3	1.00	1.00	1.00	1.00
IL2	Le4	1.00	1.00	1.00	1.00		So4	1.00	1.00	1.00	1.00
	Le5	1.00	1.00	1.00	1.00	ISo3	So5	1.00	1.00	1.00	1.00
IL3	Le6	1.00	1.00	1.00	1.00		So6	1.00	1.00	1.00	0.00 *
	Le7	0.20 *	1.00	1.00	0.00 *	ISo4	So7	1.00	1.00	1.00	0.00 *
IL4	Le8	1.00	1.00	1.00	1.00		So8	1.00	1.00	1.00	0.00 *
	Le9	1.00	1.00	1.00	1.00	ISo5	So9	1.00	1.00	1.00	0.00 *
IL5	Le10	1.00	1.00	1.00	1.00		So10	1.00	1.00	1.00	1.00
	Le11	1.00	1.00	1.00	1.00		So11	1.00	1.00	1.00	1.00
	Le12	1.00	1.00	1.00	1.00	IG1	Go1	0.00 *	1.00	1.00	0.00 *
ISt1	St1	0.00 *	1.00	1.00	0.00 *		Go2	0.00 *	1.00	1.00	0.00 *
	St2	0.80	1.00	1.00	1.00		Go3	0.00 *	1.00	1.00	0.00 *
	St3	0.20 *	0.80	1.00	0.00 *		Go4	0.00 *	1.00	1.00	0.00 *
ISt2	St4	0.80	0.80	1.00	1.00	IG2	Go5	0.00 *	0.00 *	1.00	0.00 *
	St5	0.80	1.00	1.00	1.00		Go6	0.00 *	0.00 *	1.00	0.00 *
ISt3	St6	0.80	0.80	1.00	1.00	IG3	Go7	0.00 *	0.00 *	1.00	0.00 *
	St7	1.00	0.80	1.00	0.00 *		Go8	0.00 *	0.00 *	1.00	0.00 *
	St8	0.60 *	1.00	1.00	0.00 *		Go9	0.00 *	0.00 *	1.00	0.00 *
ISt4	St9	1.00	1.00	1.00	1.00	IG4	Go10	0.00 *	1.00	0.00 *	0.00 *
	St10	1.00	1.00	1.00	1.00		Go11	0.00 *	1.00	0.00 *	0.00 *
	St11	1.00	1.00	1.00	1.00		Go12	0.00 *	1.00	0.00 *	0.00 *
IE1	En1	1.00	1.00	1.00	0.00 *	IG5	Go13	1.00	1.00	1.00	1.00
	En2	1.00	1.00	1.00	0.00 *		Go14	1.00	1.00	1.00	1.00
	En3	0.20 *	1.00	1.00	0.00 *		Go15	1.00	1.00	1.00	1.00
	En4	1.00	1.00	1.00	1.00	IP1	Pe1	1.00	1.00	1.00	0.80
	En5	0.00 *	0.40 *	1.00	0.00 *		Pe2	1.00	1.00	1.00	0.80
	En6	1.00	1.00	1.00	1.00		Pe3	1.00	1.00	1.00	0.80
	En7	1.00	0.80	1.00	1.00	IP2	Pe4	1.00	1.00	1.00	1.00
IE2	En8	1.00	1.00	1.00	0.00 *		Pe5	1.00	1.00	1.00	1.00
	En9	1.00	1.00	1.00	1.00		Pe6	1.00	1.00	1.00	1.00
	En10	0.20 *	0.80	1.00	0.00 *		Pe7	1.00	0.00 *	1.00	0.00 *
	En11	1.00	1.00	1.00	1.00		Pe8	1.00	1.00	1.00	1.00
IE3	En12	1.00	1.00	1.00	1.00		Pe9	1.00	1.00	1.00	1.00
	En13	1.00	1.00	1.00	0.00 *		Pe10	1.00	0.00 *	1.00	0.00 *
	En14	1.00	1.00	1.00	0.00 *		Pe11	0.00 *	0.00 *	1.00	0.00 *
IE4	En15	1.00	1.00	1.00	1.00		Pe12	0.00 *	0.00 *	1.00	0.00 *
	En16	1.00	1.00	1.00	0.00 *	Number of items eliminated		23	10	3	36
	En17	0.00 *	1.00	1.00	0.00 *	Number of items remaining		55	68	75	42

*: eliminated.

shows the level of agreement of the panelists regarding the suitability of each item with stakeholder groups. Items with a level of agreement < 67% were eliminated from the list. The result shows that the number of items eliminated from each stakeholder group was as follows: Students (Stu) 23 items, Staff (Sta) 10 items, Faculty Members (FM) 3 items, and Community Members (CM) 36 items. Consequently, the remaining items for each stakeholder group were 55, 68, 75, and 42, respectively.

A prototype instrument with a list of relevant questions for each stakeholder group was generated. The result showed a consensus on all items because all levels of agreement were > 67%. Therefore, the modified Delphi survey was completed.

The process continued by analyzing the content validity index, starting with calculating I-CVI and UA for each item, as shown in

Table 8. Prototype instrument for Students (55 items).

No	Code	I-CVI	UA	No	Code	I-CVI	UA	No	Code	I-CVI	UA
1	Le1	1.00	1	20	En2	1.00	1	39	So8	1.00	1
2	Le3	1.00	1	21	En4	1.00	1	40	So9	1.00	1
3	Le4	1.00	1	22	En6	1.00	1	41	So10	1.00	1
4	Le5	1.00	1	23	En7	1.00	1	42	So11	1.00	1
5	Le6	1.00	1	24	En8	1.00	1	43	Go13	1.00	1
6	Le8	1.00	1	25	En9	1.00	1	44	Go14	1.00	1
7	Le9	1.00	1	26	En11	1.00	1	45	Go15	1.00	1
8	Le10	1.00	1	27	En12	1.00	1	46	Pe1	1.00	1
9	Le11	1.00	1	28	En13	1.00	1	47	Pe2	1.00	1
10	Le12	1.00	1	29	En14	1.00	1	48	Pe3	1.00	1
11	St2	0.80	0	30	En15	1.00	1	49	Pe4	1.00	1
12	St4	0.80	0	31	En16	1.00	1	50	Pe5	1.00	1
13	St5	0.80	0	32	So1	1.00	1	51	Pe6	1.00	1
14	St6	0.80	0	33	So2	1.00	1	52	Pe7	1.00	1
15	St7	1.00	1	34	So3	1.00	1	53	Pe8	1.00	1
16	St9	1.00	1	35	So4	1.00	1	54	Pe9	1.00	1
17	St10	1.00	1	36	So5	1.00	1	55	Pe10	1.00	1
18	St11	1.00	1	37	So6	1.00	1		Sum	54.20	51
19	En1	1.00	1	38	So7	1.00	1

,

Table 9. Prototype instrument for Staff (68 items).

No	Code	I-CVI	UA	No	Code	I-CVI	UA	No	Code	I-CVI	UA
1	Le1	1.00	1	24	En1	1.00	1	47	So8	1.00	1
2	Le2	1.00	1	25	En2	1.00	1	48	So9	1.00	1
3	Le3	1.00	1	26	En3	1.00	1	49	So10	1.00	1
4	Le4	1.00	1	27	En4	1.00	1	50	So11	1.00	1
5	Le5	1.00	1	28	En6	1.00	1	51	Go1	1.00	1
6	Le6	1.00	1	29	En7	0.80	0	52	Go2	1.00	1
7	Le7	1.00	1	30	En8	1.00	1	53	Go3	1.00	1
8	Le8	1.00	1	31	En9	1.00	1	54	Go4	1.00	1
9	Le9	1.00	1	32	En10	0.80	0	55	Go10	1.00	1
10	Le10	1.00	1	33	En11	1.00	1	56	Go11	1.00	1
11	Le11	1.00	1	34	En12	1.00	1	57	Go12	1.00	1
12	Le12	1.00	1	35	En13	1.00	1	58	Go13	1.00	1
13	St1	1.00	1	36	En14	1.00	1	59	Go14	1.00	1
14	St2	1.00	1	37	En15	1.00	1	60	Go15	1.00	1
15	St3	0.80	0	38	En16	1.00	1	61	Pe1	1.00	1
16	St4	0.80	0	39	En17	1.00	1	62	Pe2	1.00	1
17	St5	1.00	1	40	So1	1.00	1	63	Pe3	1.00	1
18	St6	0.80	0	41	So2	1.00	1	64	Pe4	1.00	1
19	St7	0.80	0	42	So3	1.00	1	65	Pe5	1.00	1
20	St8	1.00	1	43	So4	1.00	1	66	Pe6	1.00	1
21	St9	1.00	1	44	So5	1.00	1	67	Pe8	1.00	1
22	St10	1.00	1	45	So6	1.00	1	68	Pe9	1.00	1
23	St11	1.00	1	46	So7	1.00	1		Sum	66.80	62

,

Table 10. Prototype instrument for Faculty Members (75 items).

No	Code	I-CVI	UA	No	Code	I-CVI	UA	No	Code	I-CVI	UA
1	Le1	1.00	1	26	En3	1.00	1	51	So11	1.00	1
2	Le2	1.00	1	27	En4	1.00	1	52	Go1	1.00	1
3	Le3	1.00	1	28	En5	1.00	1	53	Go2	1.00	1
4	Le4	1.00	1	29	En6	1.00	1	54	Go3	1.00	1
5	Le5	1.00	1	30	En7	1.00	1	55	Go4	1.00	1
6	Le6	1.00	1	31	En8	1.00	1	56	Go5	1.00	1
7	Le7	1.00	1	32	En9	1.00	1	57	Go6	1.00	1
8	Le8	1.00	1	33	En10	1.00	1	58	Go7	1.00	1
9	Le9	1.00	1	34	En11	1.00	1	59	Go8	1.00	1
10	Le10	1.00	1	35	En12	1.00	1	60	Go9	1.00	1
11	Le11	1.00	1	36	En13	1.00	1	61	Go13	1.00	1
12	Le12	1.00	1	37	En14	1.00	1	62	Go14	1.00	1
13	St1	1.00	1	38	En15	1.00	1	63	Go15	1.00	1
14	St2	1.00	1	39	En16	1.00	1	64	Pe1	1.00	1
15	St3	1.00	1	40	En17	1.00	1	65	Pe2	1.00	1
16	St4	1.00	1	41	So1	1.00	1	66	Pe3	1.00	1
17	St5	1.00	1	42	So2	1.00	1	67	Pe4	1.00	1
18	St6	1.00	1	43	So3	1.00	1	68	Pe5	1.00	1
19	St7	1.00	1	44	So4	1.00	1	69	Pe6	1.00	1
20	St8	1.00	1	45	So5	1.00	1	70	Pe7	1.00	1
21	St9	1.00	1	46	So6	1.00	1	71	Pe8	1.00	1
22	St10	1.00	1	47	So7	1.00	1	72	Pe9	1.00	1
23	St11	1.00	1	48	So8	1.00	1	73	Pe10	1.00	1
24	En1	1.00	1	49	So9	1.00	1	74	Pe11	1.00	1
25	En2	1.00	1	50	So10	1.00	1	75	Pe12	1.00	1
									Sum	75.00	75

and

Table 11. Prototype instrument for Community Members (42 items).

No	Code	I-CVI	UA	No	Code	I-CVI	UA	No	Code	I-CVI	UA
1	Le1	1.00	1	15	St9	1.00	1	29	So5	1.00	1
2	Le3	1.00	1	16	St10	1.00	1	30	So10	1.00	1
3	Le4	1.00	1	17	St11	1.00	1	31	So11	1.00	1
4	Le5	1.00	1	18	En4	1.00	1	32	Go13	1.00	1
5	Le6	1.00	1	19	En6	1.00	1	33	Go14	1.00	1
6	Le8	1.00	1	20	En7	1.00	1	34	Go15	1.00	1
7	Le9	1.00	1	21	En9	1.00	1	35	Pe1	0.80	0
8	Le10	1.00	1	22	En11	1.00	1	36	Pe2	0.80	0
9	Le11	1.00	1	23	En12	1.00	1	37	Pe3	0.80	0
10	Le12	1.00	1	24	En15	1.00	1	38	Pe4	1.00	1
11	St2	1.00	1	25	So1	1.00	1	39	Pe5	1.00	1
12	St4	1.00	1	26	So2	1.00	1	40	Pe6	1.00	1
13	St5	1.00	1	27	So3	1.00	1	41	Pe8	1.00	1
14	St6	1.00	1	28	So4	1.00	1	42	Pe9	1.00	1
									Sum	41.40	39

.

Calculations for S-CVI/Ave and S-CVI/UA were performed. Figure 2 presents a summary of the content validity index. The I-CVI of the included items ranged from 0.80 to 1.00, exceeding the recommended level of 0.78, implying that the content of each item was excellent (a). While (b) and (c) show that the result of S-CVI/Ave had a minimum value of 0.98, and the S-CVI/UA had a minimum value of 0.91, meaning that the results were above the recommended level of 0.9 and 0.8, respectively. Therefore, it could be concluded that the content validity of the whole scale was excellent.

In Stage 5, appropriate instruments were sent randomly to the group of stakeholders and received responses from 123 students, 97 staff, 80 faculty members, and 100 community members. The results of the Pearson correlation calculation were compared to the critical value in the R Product Moment Table using α = 0.05. The results are shown in

Table 12. The result of the validity test.

Stakeholder Group	N	Critical Value (Pearson Product Moment)	Pearson Correlation		p-Value
			Min	Max	Min	Max
Stu	123	0.177	0.600	0.872	0.00	0.00
Sta	97	0.2	0.568	0.866	0.00	0.00
FM	80	0.22	0.295	0.804	0.00	0.008
CM	100	0.197	0.555	0.840	0.00	0.00

. All values of the Pearson correlation were greater than the critical value. Likewise, all p-values were less than 0.05. Thus, all items were valid.

As for the reliability test, Cronbach’s Alpha results were above the critical value of 0.6 (

Table 13. The result of the reliability test.

Stakeholder Group	N of Respondents	N of Items	Cronbach’s Alpha
Stu	123	55	0.983
Sta	97	68	0.988
FM	80	75	0.973
CM	100	42	0.977

). These results were statistically convincing that the existing instruments were consistent.

5. Discussion

To the researchers’ best knowledge, this study was the first to validate an instrument regarding ESG implementation in higher education institutions through the modified Delphi study. Instruments have been explicitly compiled aimed at higher education stakeholders, namely Students, Staff, Faculty Members, and Community Members, each of which had 55, 68, 75, and 42 items, respectively.

Although there are no standards for defining expertise, having an expert panel is essential to the Delphi technique process [65]. In the current study, the expert panelists are key persons in their position level in private higher education in Indonesia. All panelists with experience managing higher education, teaching, research, and community service activities come from different backgrounds. Their competence and diversity provided enrichment to the consensus outcome.

A consensus among the expert panelists regarding the factors and indicators considered suitable to support the factors was reached. The median range of all the included indicators was 4 to 5, and the interquartile range was 0 to 1, while the level of agreement demonstrated > 0.875, indicating good consensus.

This study has contributed to higher education by identifying factors related to ESG implementation. It was concluded that Leadership, Strategies, and Performance were the associated factors. Furthermore, twenty-eight indicators have been determined that support the factors related to ESG implementation in higher education. Five indicators support the Leadership Factor: the Rector/Leader’s Leadership, Vision and Mission, Resource Input, ESG Education, and Culture and Consensus. Four indicators support the Strategies Factor: Strategy for ESG, Plan of Strategies, Funding Planning, and Execution. Four indicators support Environmental Concerns: Energy-saving (electricity and water), Waste Reduction and Recycling, Green Building and Campus, and Transportation. Five indicators support Social Concerns: People Respect, Gender and Racial Equality, Social Values Education, Care for Weak Students, and Social Relationships. Five indicators support Governance Concerns: University Autonomy, Academic Freedom, Professor Authority, Staff Empowerment, and Information Transparency. As for the Performance Factor, there are five supporting indicators: ESG Evaluation, Student Satisfaction, Staff Satisfaction, Professor Satisfaction, and Social Satisfaction.

The next stage was to compile items based on agreed indicators. Panelists were asked their opinion regarding indicators’ suitability with stakeholder groups. Items that did not receive panelist approval were eliminated. Thus, the remaining items were items that have received agreement from the panelists. The items were in the form of simple questions that the respondents easily understood; therefore, the respondents were expected to provide the proper answer accurately.

The Content Validity of this study was good. The I-CVI of the included items varied from 0.80 to 1.00, which was higher than the recommended level of 0.78, suggesting that each instrument’s content was excellent. The S-CVI/UA of each instrument was above 0.91, reaching the acceptable value of 0.8. Similarly, all of the S-CVI/Ave were above the proper value of 0.9, indicating the content validity of the whole scale was excellent. Higher education can use this instrument as a tool to monitor ESG implementation.

Although this study has been conducted very carefully, it is acknowledged that it has weaknesses and limitations. The deductive technique that supports the focus group discussion was used to collect as much information as possible from reliable sources; nonetheless, it was noticeable that this study could not cover all of them. On the other hand, the invited experts were all from the same university, which did not represent the diversity of higher education options.

Despite the limitations, this study provides essential information in the form of ready-to-use instruments. Figure 3 provides a summary of what resulted from each step in this study.

6. Conclusions

This study has established an instrument for assessing ESG implementation in higher education. Six factors were identified: Leadership, Strategies, Environmental, Social, Governance, and Performance. The six factors were latent variables requiring measurable indicators to support them. Furthermore, twenty-eight supporting indicators have been identified. These indicators were derived into seventy-eight items in the form of simple statements.

In connection with the four stakeholder groups identified as potential respondents, an in-depth study was conducted to determine the appropriate items for each stakeholder group. Finally, it was identified that the number of items varies for each group as follows Students (55 items), Staff (68 items), Faculty Members (75 items), and Community Members (42 items).

Determination of factors, indicators, and items was performed through a deductive study, focus group discussion, and consensus-seeking from a panel of experts. The panelists were key persons in higher education management. The Modified Delphi Method was applied to obtain consensus. The questions to the panelists were not open-ended but structured questions that have been compiled based on reliable previous literature. The panel of experts reached a consensus on all items, and both the individual items and overall scale were found to have excellent content validity.

The final versions of these instruments were valid and reliable. These instruments were ready to be used to conduct empirical research in higher education that will provide information on the implementation of ESG in higher education institutions. The managerial impact is to increase the interest of higher education managers in implementing ESG to support the contribution of higher education institutions to the achievement of the SDGs. This study proposes that higher education leaders consider formulating policies, especially developing strategies related to education, implementation, and evaluation related to ESG, and assessing ESG performance as a measure of stakeholder satisfaction. In practice, higher education needs to give authority to study programs and lecturers to determine the curriculum, teaching materials, and activities related to ESG and involve students to participate.

The initiation of determining factors, indicators, and items from various reliable sources might not cover all available information. Therefore, it could become the potential weakness of this study. Likewise, panelist selection was limited to one institution. It might not be able to represent the diversity of higher education institutions. Therefore, further research can be conducted by finding information from the latest research and varied panelists. Thus, it is hoped that other factors or indicators can be found to enrich the instruments. The results of this study also provide insight for further research to prove the existence of a recursive or non-recursive model among the factors.