Developing a Practical Framework of Sustainability Indicators Relevant to All Higher Education Institutions to Enable Meaningful International Rankings

Round 1

Reviewer 1 Report

The paper aims to provide national policy makers with a simple set of indicators to facilitate measuring progress towards sustainability for the University sector, within the context of national sustainability data collection efforts. The paper is very clear and well referenced in sims, structure and contents.

In section 2.2 I suggest to define each indicator to be more clear (what is greening campus? And campus ecology?)

in the conclusion express well the novelty of you study.

Author Response

In section 2.2 I suggest to define each indicator to be more clear (what is greening campus? And campus ecology?)

inserted at line 222 ".....and not because of successful implementation of campus greening, that is measures to reduce the environmental impacts associated with HEI material and energy throughput."

Campus ecology in table 2 refers to the name of an indicator framework, not an indicator

In the conclusion express well the novelty of you study.

inserted at line 381 "The novelty of this study entailed the development a simple indicator set to guide the HEI sector on its transition towards sustainability, based on consensus between the most frequently used indicator frameworks, so that many HEIs will be familiar with indicators selected."

Reviewer 2 Report

This article is on an important subject, the responsibility of higher education institutions to support the sustainability agenda, not only in terms of their own operations but also their teaching and research. It links that responsibility to the ubiquity of university 'rankings', which now have such an important role in shaping institutional strategy. 

The article reviews the elements, and their weightings, that are used to compile the most popular of these rankings (which generally do not address issues of sustainability). It then examines two of the most popular of the more focused measurement tools that do seek measure institutional performance in relation to sustainability. It concludes that neither is fully up to the task, and they propose a new index. However, much of the data needed for such an index is not yet publicly available in Ireland.

There are a number of additional points that might have been worth considering:

1. It might have improved the article if the authors had placed more stress on the role that universities can play in leading the wider public debate about sustainability - in addition to any sustainability oriented courses and research on sustainability;
2. While the authors do mention in passing the principled objections to university rankings in general (unreliable - or, at any rate - educationalist data, distortion of institutional strategies, 'gaming' the rankings, implicates for academic autonomy etc), they do not consider whether the same objections could be made to any sustainability index. A particular issue is the robustness and resilience of the data used;
3. The authors point out that much of the data needed to develop such an index in Ireland is not (yet) publicly available. They also, very briefly, elude to the position in the UK. But it would have been interesting to know how widely available the necessary data is in the rest of Europe, and indeed internationally. This would have been a major task, and beyond the capacity of the authors / scope of the article. But it would have been good to have seen at least a nod in the direction of a more comparative perspective.

Maybe the authors could reflect on these three points and, if they accept them, make adjustments to the text.

Author Response

It might have improved the article if the authors had placed more stress on the role that universities can play in leading the wider public debate about sustainability - in addition to any sustainability oriented courses and research on sustainability

This is a valid point and an interest topic for further research but the purpose of this paper was to develop indicators for the sector based on consensus among the most subscribed to HEI sustainability assessment which do not focus on leading the wider public debate and therefore beyond the scope of this research. 

While the authors do mention in passing the principled objections to university rankings in general (unreliable - or, at any rate - educationalist data, distortion of institutional strategies, 'gaming' the rankings, implicates for academic autonomy etc), they do not consider whether the same objections could be made to any sustainability index. A particular issue is the robustness and resilience of the data used

inserted at line 349 "However, despite the general pitfalls associated with scoring and ranking, which are equally applicable to HEI sustainability indexes, the widespread adoption of frameworks that use such mechanisms to gauge progress relative to other HEIs, can often be a motivating factor to promote campus greening activities."

The authors point out that much of the data needed to develop such an index in Ireland is not (yet) publicly available. They also, very briefly, elude to the position in the UK. But it would have been interesting to know how widely available the necessary data is in the rest of Europe, and indeed internationally. This would have been a major task, and beyond the capacity of the authors / scope of the article. But it would have been good to have seen at least a nod in the direction of a more comparative perspective.

This is an excellent point and I hope to do further research on the widespread coverage of the proposed indicators but its a very time consuming task.

As shown at line 170 "As of 2020, STARS had 1,016 participant HEIs registered to use the reporting tool, with 671 receiving a STARS rating, in 42 countries [68]. The UI Green Metric, established in 2010, had participation from 95 HEI in 35 countries. By 2019, 778 HEIs in 75 countries were ranked [69]. "

It is assumed that by developing an indicator set based on the two most widespread HEI sustainability indicators, the proposed indictor coverage is not unreasonable for HEIs in most countries to collect

inserted at line 334 "It should also be noted that in choosing solely the most popular HEI assessment frameworks to inform indicator selection, the proposed indicator set relies on popular consensus, rather than the expert opinion of individuals, which tends to inform a majority of relevant academic literature. This method was selected as being more likely to encourage widespread adoption of the proposed indictors as a significant number of HEIs have already shown that they are able to collect data for these indicators."

Reviewer 3 Report

This is a very well written paper that provides a summary of the very complex area of higher education rankings with an overlay of environment sustainability. 

The challenge, not as much for the paper as the reality, is that our rankings of institutional excellence depends primarily on the vantage point of the reviewer and reader. And this is the main problem. When we look at the input for these assessments, each is limited in its own way by the focus on particular inputs rather than others. This is made very clear in Table 1 starting on page 3. 

Ultimately, what is the international ranking for? When we measure a university, are we measuring their status in the world (reputations, citations), publication (citations, etc.), teaching (limited at best), or other? Where does "sustainability" play in the excellence of a university. Sure, it is important, but so is teaching and teaching is measured by faculty:student ratio, a poor measure at best. I've written and presented on these issues before, which is why I look at international rankings with a very limited view: in the end, do they really matter, beyond those who use it for international marketing? Thus, the question remains: how does sustainability factor in? It might for the Generation Z students that care more than their predecessors on these things, but are they the ones deciding where to go to top-level institutions, or is it their parents? 

Regardless, this is an interesting review of the issues. An excellent piece. 

Author Response

Thank you for the great and insightful feedback.

I think your most interesting point strikes at the heart of the sustainability issue "Thus, the question remains: how does sustainability factor in? It might for the Generation Z students that care more than their predecessors on these things, but are they the ones deciding where to go to top-level institutions, or is it their parents? "

Its easy for us to say all we have to do is change the mental models and mindsets of the next generation to solve problems that we inherited from a previous generations ways of thinking and the associated behavioural patterns driven by both conscious and unconscious factors. I have spent more time than I would like to admit thinking on this issue and have come to the preliminary conclusion that the answers to these problems need further investigation. 

Reviewer 4 Report

In the title is included “Relevant to all”. Your work does not explain about that. It is talking about “relevant national data frameworks”, “sustainability indicators in Ireland” and “academic literature", but could be necessary link different relevancies in a more strong explanation.

I am not sure that 2.1. Indicator frameworks at HEIs and 2.2. Sustainability Indicators at HEIs are part of 2. Materials and methods… Could be possible to reorganize information in order to have a better understanding of foundations and methods included, to inform readers before to plan study, and to explain how does it built his assessment? Method could explain that your team was building and validating a  

Authors say: “The objective of this paper is to assess the current extent of sustainability indicator utilisation and to outline a practical process for development of sustainability indicator sets for the HEI sector, with illustrative application to Ireland’s higher education sector”. First part of objective is descriptive. It could be better to bring to part 1 of the paper. Second part is, sensu stricto, methological, but it is not possible to find method in the paper.

Conclusion are not in abstract.

Objectives and Conclusions must be linked.

Could be it posible erase: “Reference [60] suggest that” and put “[60]” at the end of the paragraph?

It forgot header in Table 3 and Table 4.       

 

Author Response

In the title is included “Relevant to all”. Your work does not explain about that. It is talking about “relevant national data frameworks”, “sustainability indicators in Ireland” and “academic literature", but could be necessary link different relevancies in a more strong explanation.

Inserted at line 350 "The absence of ranking in the proposed framework makes it relevant to all HEIs as it does not penalise HEI based on their local attributes as most HEI sustainability frameworks do."

I am not sure that 2.1. Indicator frameworks at HEIs and 2.2. Sustainability Indicators at HEIs are part of 2. Materials and methods… Could be possible to reorganize information in order to have a better understanding of foundations and methods included, to inform readers before to plan study, and to explain how does it built his assessment? Method could explain that your team was building and validating a

2.1 and 2.2 form part of a critical review of HEI sustainability indictors which inform final indicator selection. Therefore they should stay in the materials and methods section. See response to next point to clarify the methods utilised in this paper.

Authors say: “The objective of this paper is to assess the current extent of sustainability indicator utilisation and to outline a practical process for development of sustainability indicator sets for the HEI sector, with illustrative application to Ireland’s higher education sector”. First part of objective is descriptive. It could be better to bring to part 1 of the paper. Second part is, sensu stricto, methological, but it is not possible to find method in the paper. 

The method for the paper is outlined directly after the objective in line 83 "A critical assessment of selected existing international indicator frameworks was undertaken to identify the most frequently adopted sustainability indicator frameworks in use. Drawing on the findings of this review, an indicator set for the HEI sector is described based on indicator similarities identified between the UIGreenMetric and STARS. The proposed indicator set is then assessed against data availability by comparison with current data collection methods adopted by HEIs in Ireland."

Conclusion are not in abstract. Objectives and Conclusions must be linked.

The main points of the conclusion are included in the abstract namely an indicator set was developed based on international consensus with potential international application, and then compared to data collection efforts in Ireland which found major data gaps.

The objectives are linked to the conclusions. See how objective above and line 83 above compare to line 379 in conclusions "Existing international frameworks measure different aspects of sustainability, are not using optimal metrics, so for these reasons are not satisfactory for all HEIs to adopt for monitoring their transition to greater sustainability. The analysis of UIGreenMetric and STARS metrics, revealed weaknesses in the indicators selected and metrics adopted opening up the likelihood of double counting due to ranking and scoring of indicators. The novelty of this study entailed the development of a simple indicator set to guide the HEI sector on its transition towards sustainability, based on consensus between the most frequently used indicator frameworks, so that many HEIs will be familiar with indicators selected. International adoption of this framework may be expected to support more objective evaluation of the performance of HEIs, and result in more transparent national and international comparisons of HEI performance in supporting national and international actions in transitioning to a more sustainable future. Significant data gaps were found in relation to some indicators, and there was a poor match between data gathered by HEIs and those needed to monitor progress on meeting Irish government targets on enhancing sustainability."

Could be it posible erase: “Reference [60] suggest that” and put “[60]” at the end of the paragraph?

Reworded line 137: "It is suggested that ‘being green’ is increasingly impacting global university rankings, and that enhancing environmental sustainability can serve as a competitive advantage [60]."

It forgot header in Table 3 and Table 4. 

Inserted header table 3: "Weighting of criteria for UI Green Metric and STARS."

Inserted header table 4: "Simplified Sustainability Indicator Framework for HEI sector, and responsible Irish Data Collection Agencies."

 

 

Round 2

Reviewer 4 Report

The quality of this ms has been improved.