Can We Improve Indicator Design for Complex Sustainable Development Goals? A Comparison of a Values-Based and Conventional Approach

Abstract

A conceptual framework was constructed for United Nations’ complex Sustainable Development Goal (SDG) Target 4.7 focusing on education for sustainable development (ESD), and used to analyse the usefulness and character of indicators produced from a values-based approach called ESDinds, compared to a UN process. The analysis shows that the latter generated very few indicators concerning the wider aspects of knowledge such as ‘critical thinking’ or ‘learning to learn’. The values-based approach, created for a different purpose, produced complementary if not better coverage of Target 4.7, including finely-developed concepts for competencies and less tangible aspects. It is suggested that the UN process would benefit from ESDinds design elements such as intersubjective and slightly disruptive elements, purposeful contextualisation at group level, and a holistic and inductive consideration of values. The use of a reference ‘fuzzy framework’ of slightly generalised proto-indicators suited for deep contextualisation locally is recommended, rather than any rigid global-level indicator with unclear local value. It is recommended that ESD practitioners immediately develop localised interpretations of valid measures for whatever final Target 4.7 indicator is selected by the UN, as this localisation process will itself cause important learning towards local ESD achievements.

1. Introduction

The creation of the 2030 Sustainable Development Goals (SDGs) responded to the call by United-Nations Secretary-General Ban Ki-Moon for “the most inclusive and transparent development agenda the world has ever seen” ([1], p. 4). People in 194 countries contributed to the goals through a systematic process of 88 national consultations, 11 thematic dialogues, and a global online survey with over 8.5 million participants [1,2]. The resulting goals are also distinctive in being applicable to all nations, regardless of gross domestic product or geographical location [3] which makes them a marked improvement on the Millennium Development Goals, which were widely criticised for ‘ghettoising’ development as an issue for the global South ([4], p. 1184). The SDGs represent the first explicit acknowledgement, at the level of global goals, of the interconnectedness of the challenges surrounding sustainability and the corresponding need for integrated problem-solving [3].

While the SDGs are in many ways an important accomplishment, fundamental questions remain around how their implementation will be monitored and evaluated at all levels, from the global to the most immediately local. It is difficult to overstate the importance of identifying relevant and valid indicators. The 2015 Millennium Development Goals (MDGs) Report has explicitly acknowledged that “what gets measured gets done” (p. 10, [5,6]). The choice of indicators to measure progress towards the 169 SDG targets will contribute substantially to shaping national policies, strategies and action plans in most UN Member States, from now until 2030.

As noted by Gudmundsson [7] and subsequent authors [8,9,10,11], the contributions made by indicators to sustainable development go far beyond their instrumental uses in providing data that may inform policy formation. In addition to their significance in shaping conduct, their mere existence alters awareness, shapes attitudes and directs resources for the justification of decisions. These symbolic implications of indicators may be found entirely independent of, and even in conflict with, the empirical data which the indicators generate [8]. As an example, the use of gross domestic product (GDP) as a symbolic indicator of economic growth may be used to justify inaction or ‘business-as-usual’ policies that can directly block the achievement of ecologically-oriented goals [12].

The MDGs report calls for a “data revolution” through the use of the phrase “together we can measure what we treasure” ([5], p. 13). This recognises the intimate relationship between goals, the values from which they spring, the indicators by which they are recognised, and the tools selected for their assessment. While deeply evocative, the use of the first person plural in the statement ‘we can measure what we treasure’ implies singular identity and vision. This rhetorically compelling assertion erases the diversity of values and ideologies (at times incompatible) that characterise discussions bearing on sustainable development (SD) within institutions, whether global or local. Hopwood, for example, maps more than 30 different views on SD within a two-dimensional space—positioning them from ‘low environmental concern’ through ‘techno-centred’ to ‘eco-centred’ on the horizontal axis, and from low to high concern for socio-economic justice on the vertical axis [13]. Thus although the need is acknowledged for indicators to accommodate plural viewpoints, to measure what ‘matters’ and to involve all those willing to participate, there is no clear pathway yet known to do this. The UN’s approach may be the best under the circumstances, but it does not claim to be optimally designed.

In contemplating the many possible starting points towards designing effective working measures for the SDGs, the authors brought in a further dimension: the ‘missing’ fourth pillar. Without endorsing a particular definition of SD, we note assumptions and biases that the UN approach appears to perpetuate, which contribute to the marginalisation of those dimensions of sustainability that are not found within the established three dimensions of the economic, the environmental and the social. Any developments constrained to those three dimensions are insensitive, at minimum, to cultural/aesthetic dimensions, e.g., general discussions of cultural integrity and vitality [14,15,16] and specific discussions in relation to Indigenous communities [17]; the role of the arts in sustainability [18]; political/institutional dimensions, e.g., `good governance’ [19,20]; and religious/spiritual dimensions [21,22,23]. While in many ways these excluded dimensions are mutually distinct, they intersect in their shared interest in the category of values, in whatever manner these are understood. Although the domain of values certainly does not cover the full range of those excluded dimensions, it provides an entry point for bringing those dimensions into discussion with the three ‘pillars’ that appear to be more amenable to traditional forms of conception and measurement [24].

Values have historically been excluded from the realm of indicator development because they are perceived as intangible and difficult to measure [24,25,26]. However, it is possible to operationalise values when they are conceptualised intersubjectively [27]. That is, rather than viewing values as static latent traits possessed by individuals, they may become visible in groups when operationalised through systematic processes in clearly defined practical contexts, such as their normal working practice. An intersubjective approach to the design of indicators for values has recently been piloted in a variety of settings [24,27,28,29,30,31,32,33,34,35,36], including for formal evaluation. It typically involves an intersubjective process of values elicitation at the level of short statements about what is ‘valuable, worthwhile and meaningful’ to the group, e.g., “Mistakes are understood as opportunities to learn” coupled with gently disruptive probing to force conscious recognition of tacit knowledge, to denaturalise socially indicated responses, and finally to agree on local indicators. This inductive values- based approach to developing indicators differs from the conventional one used by the UN in that it starts with an open question (“What is ‘important’ to you about your group work?”); it proceeds intersubjectively; and it is purposely disruptive in ways that provoke discussion and contributions from all members until some resolution is reached. The indicators were generated by working groups (‘doers’) relating to their work/practice (‘doing’), rather than by individual leaders or national or regional representatives in an abstract and intellectual mode.

In this paper the authors present an exploratory study of the indicators produced via this values-based approach compared to the conventional approach used by the UN in order to test their influences to the operationalisation of complex SDG targets. To do this the values-based indicators developed by twenty-one organisations to evaluate their broadly-defined ‘ESD’ programs are compared to those obtained through the UN process for SDG Target 4.7 which is focused on ESD. The analysis involved first constructing a conceptual framework for Target 4.7 using the method of Hak et al. [37,38], and using it to systematically compare, via coding, the two sets of indicators for their usefulness for operationalising in terms of their potential relevance (item validity), and coverage (sampling validity).

The following sections discuss principles of indicator design, as background to the subsequent description of the conventional approach currently used within the UN SDG process and the values-based approach that we have studied.

1.1. Emerging Principles of Indicator Design

The development of indicators for the SDGs should be systematic and informed by theory [37,38] and should be undertaken in a way that goes beyond ‘what can be measured using currently available methods’ in order to measure ‘what should be measured’ ([39], p. 295, p. 296). The strong appeal by McCool and Stankey that we go beyond the convenient [39] constitutes recognition that the development of indicators is informed by both social and political negotiations [40]. Were indicator development purely technical, there would be no risk that the convenient could be substituted for the valid.

Hák et al. [37] advocate the creation of coherent conceptual frameworks for SDG targets, as a necessary condition for ensuring the validity of linkages between indicators and the facts they are to represent. If the conceptual framework behind SDG targets is not internally coherent, that incoherence will compromise their indicators (see also [40]). This echoes pre-SDG indicator development guidelines such as those included by Dalal-Clayton and Bass in their resource book Sustainable Development Strategies [41], advocating the creation of a ‘framework of parts and aims’ as the first essential step in designing an indicator-based assessment. In this context, ‘parts’ are understood as dimensions, elements or themes, and ‘aims’ as goals, objectives, principles or criteria. These authors suggest that using a framework of parts and aims as the foundation for indicator development has numerous advantages, e.g., avoidance of measuring the same part twice or omitting essential parts; highlighting parts for which no measurable indicator exists; and showing the relationships between different parts and appropriate weightings for each of them in the development of indices (see also [7,8]).

Dalal-Clayton and Bass [41] further recommend the development of a framework of indicators that is systemic (organised to illustrate key features of the system and subsystems), hierarchical in the sense that the parts are organised into a series of levels, logical in the sense of being a series of means and ends (e.g., ‘we measure A by measuring B, B by measuring C…’), and readily communicable to non-specialists. More recently, Lozano, Llobet and Tideswell [42] have emphasised the importance of examining the interlinkages between different indicators within the framework.

Assuming that a given indicator is empirically defensible, it must also be recognised as socially relevant (e.g., policy relevance, relevance to specific stakeholder groups, or applicability at the appropriate level) [37]. Optimising indicators for both the empirical and the social, in turn, requires clear conceptions of key terms such as ‘sustainable’, ‘efficient’ or ‘substantial’ [38]—terms whose ambiguity may have been useful and intentional when the targets were negotiated, but which then pose challenges for operationalisation. These issues can be avoided in narrowly defined SDG areas, but are problematic with complex areas which involve intermeshed concepts, such as ESD.

1.2. The Conventional UN Approach to Developing SDG Indicators

The task of deciding what should be measured to evaluate the SDGs at a global level was delegated to the ‘Inter-Agency and Expert Group on Sustainable Development Goal Indicators’ (IAEG-SDG) by the United Nations Statistical Commission in March 2015. This group consists of representatives from the national statistical offices of 28 ‘member’ countries (consisting of seven each from Africa and Asia, two each from Oceania and the Caribbean, one from North America, three from Central and South America, and five from Europe) [43], as well as observers from non-member countries, regional commissions, and regional and international agencies (e.g., non-governmental organisations) who can provide technical advice and support [44].

In its March 2016 report, demonstrating the extent to which indicator development is as much socio-political as it is empirical, the IAEG-SDG describes the initial step in the process of indicator development as involving an “open consultation… with all countries, regional and international agencies, civil society, academia and the private sector” from August to September 2015 ([44], p. 6). This was followed by an initial coding of all proposed indicators as either ‘green’ (general agreement) or ‘grey’ (more discussion required) by the panel. Two limited public consultation processes were then launched—the first, on the ‘green’ indicators, open for only three days in November 2015 [45], and the second, on the ‘grey’ indicators, for seven days in December 2015 [46]. While the consultations received little or no attention in the mainstream media and academic journals, their output represents a snapshot of the international discourse surrounding SDG indicators at the crucial launch time of the goals. They are noteworthy not only in the sense of who has contributed and what has been said, but also—as we discuss below—because of what and who has not been included.

It is apparent from the March 2016 report and related IAEG-SDG documents [44,45,46] that the IAEG-SDG approach to indicator design did not begin with the decomposition of the target into a framework of parts and aims, as advocated by Dalal-Clayton and Bass [41]. Rather, individuals and organisations were invited to comment on draft indicator proposals issued by the IAEG-SDG, and submit their recommendations for alternatives. This raises the possibility that proposed indicators might have been informed as much by participants’ cultural backgrounds, institutional affiliations and interests as by the functional need. Insofar as there is no opportunity for critical cross-examination or acknowledgement of the full range of concepts to be covered, the entire process invites contributors to formalise and uncritically advance their own perhaps entrenched ways of thinking about what constitutes ‘SD’.

Operationalizing ‘ESD’ through the Conventional Approach

The strategic importance of education as a main Sustainable Development Goal (Goal 4) has been widely acknowledged, and its two-way linkages with other goals outlined, in numerous United Nations flagship reports—to the extent that the goals can be implicitly organised as a conceptual map with education at the centre [47].

In this paper we focus specifically on operationalising Target 4.7, the target which relates most explicitly to education for sustainable development (ESD):

SDG Target 4.7:

“By 2030, ensure that all learners acquire the knowledge and skills needed to promote sustainable development, including, among others, through education for sustainable development and sustainable lifestyles, human rights, gender equality, promotion of a culture of peace and non-violence, global citizenship and appreciation of cultural diversity and of culture’s contribution to sustainable development.”

We have chosen to focus on this SDG target because its great breadth provides an excellent overlap of subject domain with that available from the values-based approach that we wish to consider. In examining Target 4.7 we do not adopt a position on what is meant by either ‘sustainable development’ or ‘education for sustainable development’ and we fully acknowledge that the term ‘ESD’ is highly controversial and that its usefulness has been contested, especially by environmental educators [48]. Nevertheless, we accept that the language of ESD has become so entrenched—not only in the SDGs themselves but also in a vast body of United Nations literature, particularly around the UNESCO 2004–2014 ‘Decade of Education for Sustainable Development’—that it would be unrealistic to imagine that current objections alone could secure its replacement with a less politically loaded term (such as ‘education for sustainability’). We expand further on these points, with relevant supporting literature, in Appendix A.

While some targets for other SDGs also have relevance to ESD, especially Target 12.8 (“ensure that people everywhere have the relevant information and awareness for sustainable development and lifestyles in harmony with nature”) and Target 13.3 (“improve education, awareness-raising and human and institutional capacity on climate change mitigation, adaptation, impact reduction and early warning”), these are very much narrower in focus than Target 4.7. The complexity of Target 4.7 not only provides better domain overlap for our comparison (see Section 1.3 below), but also epitomises both the challenges of, and the crucial need for, a systematic and theoretically-grounded approach to conceptualisation and operationalisation in the development of complex SDG indicators.

The initial attempt by the IAEG-SDG [44] to operationalise this target, using one indicator, took a very narrow interpretation that focused specifically on scientific knowledge:

Candidate Indicator (UN-IAEG) for Target 4.7:

“Percentage of 15-year old students enrolled in secondary school demonstrating at least a fixed level of knowledge across a selection of topics in environmental science and geoscience. The exact choice/range of topic will depend on the survey or assessment in which the indicator is collected (Dis-aggregations: sex and location (and others where data are available).”

This indicator is problematic not only because of the narrow scope, but also its promotion of a transmissive rather than a transformative view of ESD (c.f. [48]). The recognition by the IAEG members that this indicator does not fully reflect the concepts found in the target (i.e., lacks sampling validity) is evident in their decision to call for proposals for alternative indicators [46].

A total of 114 separate responses to the public consultation were received for Target 4.7, of which 83 came from civil society organisations (CSOs), mainly non-governmental organisations; 16 from United Nations agencies or national UN-related organisations; nine from national statistical offices; five from universities; and one from a working group specifically convened for the purpose of reflecting on SDG indicators (the ‘Adolescent Girl and SDG Indicators Working Group’, which was comprised of representatives from the UN Foundation and six CSOs). Several were replicated identically, but the 71 unique indicators form one set for our comparison and we label them the ‘IAEG-SDG’ responses set, produced through the conventional, UN, approach. Below we analyse their relevance and coverage for Target 4.7.

1.3. Values-Based Approaches to Indicator Design

The second approach and indicator set that we will analyse originated with the ESDinds project (ESDinds: Development of Values-Based Indicators and Assessment Tools for Civil Society Organisations Promoting Education for Sustainable Development), a project funded by the European Commission’s Seventh Framework Programme from 2008 to 2011 under the specialist funding scheme ‘Research for the Benefit of Specific Groups: Civil Society Organisations’ (BSG-CSO) [49]. The two-year project was constructed as a consortium of two academic partners (university research groups) and four civil society organisations promoting ESD in non-formal contexts:

(a)

the Alliance of Religions and Conservation, a secular organisation working with 11 major faiths;

(b)

the Earth Charter Initiative, a global network of organisations promoting the principles of the Earth Charter, with its secretariat in Costa Rica and strong representation from the global South;

(c)

EBBF (ethical business building the future), a global learning community initially registered as a non-governmental organisation in France;

(d)

People’s Theater (sic), a small German organisation.

The consortium evolved from earlier collaborations and partnerships, and consisted of organisations that viewed their ESD provision as broadly ‘values-based’—defining ‘values’ in the sense of “principles or standards of behaviour”, i.e., ethical, spiritual or moral values, as well as “[people’s] judgement of what is important in life” [50]. The overarching aim was to develop project-level indicators and tools that would not only capture the values and priorities of the participating civil society organisations (CSOs) in their work, but also help them to identify, evaluate and communicate less tangible ESD dimensions. While the ESDinds process did not set out to produce indicators that covered Target 4.7, we show that its execution within organisations with express interest in ESD was sufficient to do this (see

Table 1. A summary of factors in the two approaches which indicate domain differences or similarities for consideration in the comparison analysis.

	Values-Based Approach (ESDinds) (N = 125)		UN-IAEG Conventional Approach (N = 114)
Who was asked?	Members of 6 groups, civil society organisations (CSOs): their answers were used to construct the set		All countries; regional and international agencies, civil societies, academia and the private sector were asked.
	Members of a further 15 CSOs: their answers were used to reduce the set by clustering overlapping responses
Who answered	As above		114 groups answered, of which 83 CSOs, 16 UN Agencies or related, nine national statistics offices, five universities and one specific working group.
Question responded to	“What is worthwhile, valuable and meaningful to you about your work?”		“Please examine the Candidate Indicator given for SDG Target 4.7 and propose any alternatives you think might be more appropriate.”
Boundaries which are contextually implied:	The question was with respect to		The question was with respect to
	a	A local group	a	Mostly national or international groups
	b	Non-formal education	b	Formal or non-formal—not specified
	c	Assessment of ESD of a collective	c	Assessment of ESD of individuals
	d	Local SD focus	d	Local or global—not specified
	e	The present situation and near future	e	Present, near or far future—not specified
	f	Practice (as opposed to knowledge)	f	Practice or knowledge—not specified
	g	Holistic approach to assessment	g	Non-holistic approach to assessment
	h	ESD decomposition via local values	h	ESD deconstruction via outputs
	i	Obtained intersubjectively within the group	i	Process within groups unknown but likely to be non-inter-subjectively

).

The methodology of the ESDinds project is described fully in the final report [51] and by Podger et al. [52] and is usefully understood to be an example of ‘Research through Design’ [28]. In brief, the initial process of developing draft indicators used a multiple case study approach, incorporating diverse methods of data collection—key informant semi-structured interviews, informal interviews, participant observation, and document collection—to explore what group participants found valuable, meaningful and worthwhile in the context of their values-based ESD projects. Six organisations contributed in this way, all affiliated to the four partner CSOs. Thematic content analysis of these large data sets was based around a codebook derived from values and indicators literature, and involved tests for inter-rater reliability and member checking within all participating CSOs. During the analysis, the wording of each draft indicator was negotiated so that participating organisations found them to be comprehensible, measurable, relevant and locally valid, while attempting to achieve a level of abstraction that would make them generalisable to other contexts. This was in accordance with the ‘bottom-up and top-down’ approach presented as an example of good practice in indicator development by Reed, Fraser and colleagues [53,54,55,56,57]. The draft indicators from the different organisations were then clustered intersubjectively to produce a set of 177. This was later reduced to a final reference set of 125 applicable to multiple contexts, including but not limited to non-formal ESD [58], via field trials in 15 further organisations, and that set is centred on ESD practice. We thus analyse it for relevance and coverage of Target 4.7, denoting it the ESDinds set, and the approach as ‘values-based’. This process produced not rigid indicators, but ‘proto-indicators’—reflecting the concept of a `prototype’ in design literature, to refer to concise statements that can serve as templates or triggers for the local development of immediately relevant measurable indicators [33].

Proto-indicator sets have since been developed in a simplified manner for other contexts, including secondary schools [28,30,31,32], universities [29,59] and community-university research partnerships [34], All the sets have significant overlap, and their indicators reflect the values-based approach. However, none of those have yet achieved the same level of validation through field testing as the ESDinds set.

1.4. The Overlap of the Two Domains

The aim of this study is to compare indicators produced from the two approaches to explore their intrinsic differences with respect to useful operationalisation of Target 4.7. Ideally, the two would be developed in parallel, but without time and funds to do this we have made use of what is available, and noted areas where comparisons would not be appropriate or need extra caution. Table 1 summarises these, and is referred to in the subsequent design of analysis, presented in Section 2.2.2. The aim is to identify aspects of the indicator sets or the processes by which they are generated which suggest that further research would be useful.

2. Methods

In this section we briefly outline the approach used to develop a conceptual framework for Target 4.7 and then describe the analysis conducted to compare the IAEG-SDG and ESDinds indicators, from the conventional and values-based approaches respectively, in terms of their usefulness to the operationalisation of this target. Reference is made to their relevance and coverage of different components which would impact on the overall item validity and sampling validity of any final indicator(s) developed.

2.1. Creating a Conceptual Framework for Target 4.7

The process of creating a conceptual framework for Target 4.7 proceeded in three stages: (1) defining two sub-targets; (2) subdividing the skills sub-target, by identifying broad categories of sustainability competencies derived from academic and professional literature; and (3) identifying smaller clusters of skill competencies within those categories where applicable, again with direct reference to literature. A broad overview is given below and further details in Appendix B.

2.1.1. Top Level: Sub-Targets

Hak et al. ([37], p. 570) provide an example of the process of defining sub-targets by breaking down Target 12.3, “By 2030 achieve sustainable management and efficient use of natural resources”, into two sub-targets, “sustainable management of natural resources” and “efficient use of natural resources”. Each of these, in turn, can be subdivided, e.g., efficient use of raw materials (comprising fuels, minerals, metals and biomass), efficient use of food, efficient use of water, and efficient use of land. With each recursively created level, the sum of all the sub-targets constitutes the super-ordinate.

The application of the same approach to Target 4.7 is frustrated by the lack of natural points where it may be cleanly separated into mutually exclusive and collectively exhaustive sub-targets. In principle, the target can be split into two overall dimensions relating to learning outcomes—all learners acquire the knowledge needed to promote sustainable development, and all learners acquire the skills needed to promote sustainable development.

• Sub-Target 1: All learners and educators involved in organised learning activities, both formal and non-formal, acquire knowledge needed to promote sustainable development.
• Sub-Target 2: All learners and educators involved in organised learning activities, both formal and non-formal, acquire skills needed to promote sustainable development.

A case could also be made for including a third or alternative sub-target that can be operationalised as multiple process indicators, e.g., “All learners enrolled in formal and non-formal organised learning activities receive education for sustainable development and sustainable lifestyles” (or human rights, gender equality, peace and non-violence, etc.), on the grounds that in certain countries it may be logistically difficult or impossible to assess learning outcomes for ESD. Where this is the case, the only way of collecting meaningful data would be to focus on the provision of relevant learning activities. In this paper, however, we focus on operationalising the target as it is currently worded, with its focus on learning outcomes, only returning to this alternative sub-target in the discussion.

2.1.2. Second Level Categories: Competencies

We realised there was no need to conceptualise in great detail `knowledge that enables people to promote sustainable development’ since actions are local and contexts vary, resulting in the knowledge required to promote ‘SD’ (however it might be defined) being idiosyncratic to each locality. Attempting to specifically catalogue this knowledge for a global level of application would be counter-productive, hence the need for localisable assessments such as the ‘Sustainability Literacy Test’ (SuLiTest) developed by UNEP and other agencies. On the other hand, this same SuLiTest is helpful as an initial proxy measure for SD-related knowledge at a general level, and we advocate its use as a starting point even though it has significant limitations, focuses on a minimum level of basic knowledge, and requires further required to develop higher-level assessments [60].

In contrast, we are able to develop great detail in the conceptualisation of skills for Sub Target 2. Despite being largely overlooked by the IAEG-SDG in their indicator development process, the skills dimension features prominently in ESD in both professional and academic literature on ESD—usually framed in terms of ‘sustainability competencies’. We thus expanded our framework in the skills dimension through a sub-study of significant works relating to skills that enable people to promote sustainable development. Ultimately we produced a hybrid framework which covered the key aspects of the competencies discussed elsewhere. It was comprised of two accepted conceptual frameworks, one from professional literature [61,62] and the other from academic literature [63]. relating to Both are framed in terms of ‘sustainability competencies’, also referred to as ‘ESD competencies’ (although we note that the term ‘competency’ as defined by Rychen and Salganik ([64], p. 51) includes not only skills but also other cognitive and non-cognitive components, such as attitudes). Appendix A sets out the justification for, and caveats associated with, the use of the competencies lens (and of these specific references) within a broader literature review on ESD, while Appendix B provides further detail on the actual construction of our conceptual framework. The key concepts of the two frameworks used are summarised below:

UNESCO/UNECE model: A seminal report to the United Nations Educational, Scientific and Cultural Organisation (UNESCO) by the International Commission on Education for the Twenty-First Century, chaired by the former European Commission President Jacques Delors, distinguishes four `pillars’ for education in the twenty-first century ([61,62,63,64,65,66,67]). To these four pillars, which have been applied specifically to ESD as the foundation of the United Nations Economic Commission for Europe (UNECE) report Learning for the Future: Competencies in Education for Sustainable Development [61], a fifth was later added by UNESCO (see Appendix A), resulting in the following set of competencies [62]:

• learning to know (e.g., learning to learn, developing critical thinking, acquiring tools for understanding the world, and understanding sustainability concepts and issues);
• learning to do (e.g., acquiring technical and professional training and applying learned knowledge in daily life);
• learning to be (e.g., seeing oneself as the main actor in defining positive future outcomes, developing self-identity and self-knowledge, acting with autonomy and personal responsibility);
• learning to live together (e.g., understanding other people’s values and traditions, cooperating with people, celebrating diversity and coping with conflict);
• learning to transform oneself and society (e.g., building non-discriminatory societies, reducing ecological footprint, integrating sustainable lifestyles and promoting social solidarity) [62].

Wiek model: Through a systematic literature review, Wiek et al. [63] identify five broad categories of key competencies for sustainability:

• systems thinking competence, the ability to analyse the dynamics of complex social-ecological systems;
• anticipatory competence, i.e., the ability to create, analyse and evaluate what they term ‘rich pictures’ of the future;
• normative competence, also referred to as value(s)-focused thinking, which focuses on “the ability to collectively map, specify, apply, reconcile, and negotiate sustainability values, principles, goals, and targets” ([63], p. 10);
• strategic competence, which ensures that learning is translated into effective policies, programs and action plans; and
• interpersonal competence, which refers to the ability to motivate, enable and facilitate participatory sustainability research and collaborative problem-solving, to celebrate diversity, and critically evaluate different positions and perspectives. This is regarded as a cross-cutting issue, as the involvement of multiple actors in sustainability problems makes it essential for collective strategies to be employed in trying to solve them.

While the literature on competencies in ESD reveals a complex, multifaceted and rather low-consensus picture comprising both affective and cognitive elements (see Appendix A), we find that most of this complexity is covered within the two models described by UNESCO/UNECE and Wiek et al., respectively. For the practical purpose of this paper, which requires a framework adequate to roughly compare potential relevance (item validity) and coverage (sampling validity) of indicators, we have combined the main concepts of these two models into a single eight-category framework of ESD competencies, described in detail in Appendix B and illustrated in Figure 1, (1)–(8) below.

2.1.3. Third Level: Specific Clusters of Competencies

Of the eight broad competency categories, five contained further distinctive sub-dimensions, as shown in the bottom half of Figure 1, closely following the text of a UNECE infographic [68]. In order to balance rigour (in the sense of including a sufficient number of codes to provide a valid and reliable recognition of all analytically relevant data) with usability, we worked at the level of clusters. Our conceptual framework for Target 4.7 thus consisted of a Sub-Target for Knowledge with only one level of division of concepts, and a Sub-Target for Skills with three levels of concepts of competencies (which are shown in Figure 1).

2.2. Exploring the Usefulness of the Operationalisation of Target 4.7 through Two Contrasting Approaches

In considering how to compare the outputs from the two approaches for generating indicators, and to identify important aspects which can feed into future research design, attention must be paid to the incomplete overlap of approach domains. Had the aim of this work been to evaluate two approaches across equivalent domains, we could have done tests of relevance (item validity) and coverage (sampling validity) of the two sets of indicators using statistics. However, the authors realised it was more useful to obtain a detailed and rich overview of interesting aspects of not only the indicators but also the processes which produced them. For example, although we have described the non-conventional approach taken by ESDinds as ‘values-based’, it also has clear aspects of intersubjectivity, holism and local-level operationalisation, which might prove more important than the values lens itself. Similarly the IAEG-SDG process could be described as involving individual representatives of organisations, rather than working groups of members; direct copying of indicator proposals by other respondents; and a stronger motivation for the participating organisations to push political agendas, such as sexuality or HIV/AIDS education, rather than focusing on the specific Target 4.7.

Another consideration was whether to focus on ‘ideal single indicators’ which could potentially cover the entire remit of Target 4.7, or the combined contribution of the output set of indicators from each approach in terms of its ‘helpfulness towards operationalisation’. Since the two approaches favoured these differently, both were taken forward so as not to bias either: both provide pathways to producing good coverage.

There were 110 responding organisations in the IAEG-SDG process, providing a total of 114 indicator proposals, but a number of these proposals were repeated multiple times. Our approach required considering only the 71 unique indicators (retaining those with small variations in wording as separate items). The ESDinds set contained 125 proto-indicators which had already been clustered from others, and thus were already unique. Due to the large number of indicators in each set (71 and 125 respectively), and the complexity of the conceptual framework for Target 4.7 (see Figure 1), we developed a coding and scoring system to evaluate relevance and coverage for both individual indicators and sets of indicators. We outline this below, and then use the summary

Table 2. Scoring method used across the conceptual framework for Sub-Target 1 (Knowledge) of Sustainable Development Goals (SDG) Target 4.7 for relevance and coverage. Note that this Sub-Target is not covered in the values-based approach (ESDinds).

	Relevance	Coverage
Holistic knowledge
OR	1 point	2 points
The Sustainability Literacy Test
CONTENT x8:	1 point if AT LEAST ONE content area is assessed	1 point per content area judged to be PARTIALLY covered and 2 points per content area judged to be SYSTEMATICALLY OR RIGOUROUSLY covered
1. Education for Sustainable Development
2. Education for sustainable lifestyles
3. Human rights
4. Gender equality
5. Culture of peace and non-violence
6. Global citizenship
7. Appreciation of cultural diversity
8. Culture’s contribution to sustainable development
NONE of the above	0 points	0 points

to illustrate where less direct considerations must be used, where a lack of domain overlap occurs.

2.2.1. Codebook Development and Scoring Methodology of Content Analysis

In order to facilitate the content analysis of both datasets, we developed a codebook from the conceptual framework outlined above. The scoring systems are given in Table 2 and

Table 3. Scoring method used across the conceptual framework for Sub-Target 2 (Skills) of SDG Target 4.7 for relevance and coverage. The coloured area indicates domains not covered in the values-based approach case- studied in this work (Anticipatory Competencies).

	Relevance	Coverage
	1 point awarded	1 point per competency domain that is PARTIALLY covered, i.e., the indicator covers AT LEAST ONE of its sub-clusters
COMPETENCIES x8:	if ANY of these competency domains are assessed
L2D Learning to do/Strategic Competencies		2 points per competency domain that is SYSTEMATICALLY OR RIGOROUSLY covered, i.e., the indicator covers ALL of its sub-clusters
L2K Learning to know
L2B Learning to be
L2T Learning to transform oneself and society
L2L Learning to live together/Interpersonal
Normative Competencies	1 point awarded if ANY of these competencies are assessed (only if not already awarded for UNECE competencies)	1 point per competency domain judged to be PARTIALLY covered *.
Anticipatory Competencies
Systems Thinking Competencies		2 points per competency domain that the coder judges to be SYSTEMATICALLY OR RIGOROUSLY covered *
NONE of the above	0 points	0 points

* According to the description by Wiek et al.

. By comparing the content of each indicator proposal (IAEG-SDG) or proto-indicator (ESDinds) text with the text in the codebook, an initial subjective judgement of fit was made by the first author, and the coded datasets were subsequently checked by the other authors.

We used the codebook to analyse the relevance of individual indicators for Sub-Target 1 (Knowledge) and Sub-Target 2 (Skills) respectively, and then to explore the coverage of sub-targets, both at the level of individual indicators and across each dataset as a whole:

• In evaluating relevance for Sub-Target 1 (Knowledge) at the level of individual indicators, we scored an indicator as 1 (‘relevant’) if it explicitly measured the acquisition of knowledge in relation to either (a) at least one of the eight content areas mentioned in the target; or (b) the Sustainability Literacy Test. For this purpose, the terms ‘understanding’ and ‘proficiency’ were taken as synonymous with ‘knowledge’. Indicators that measured knowledge acquisition only in the limited sense of environmental science and/or geoscience (as included in the original indicator proposal, which the IAEG-SDG had already acknowledged as inadequate for operationalising the target) and those that did not measure knowledge acquisition at all were scored as 0 (‘not relevant’) for Sub-Target 1. Table 2 illustrates and summarises the scoring methods.
• To evaluate relevance for Sub-Target 2 (Skills) at the level of individual indicators, ‘proficiency’ was not taken as synonymous with ‘skills’, but the word ‘skills’ was not essential in itself for scoring as 1. Rather, the coder made a judgement, on the basis of the whole indicator text, as to whether the indicator would measure the acquisition of skills in one or more competency domains. Indicators measuring related aspects—such as the creation of appropriate environments for skill acquisition, the existence of policies or structures intended to support ESD skills development, or the percentage of schools providing a certain type of education—were not scored as 1 unless there was clear evidence that the indicator requires the measurement of skill-related learning outcomes for specific individuals or groups in those settings.
• In our evaluation of coverage of Sub-Targets at the level of individual indicators, we highlighted the specific content areas (for Sub-Target 1: Knowledge) or competency domains and sub-domains (for Sub-Target 2: Skills) that were hit by each indicator, scoring 1 for ‘yes’ and 0 for ‘no’ in each case. (Although there is no particular reason or advantage for a given indicator to score against more than one category, those scoring highly might be considered more potentially useful as a main indicator for Target 4.7, whereas others might have low coverage but perhaps provide innovation in how to deal with a specific and potentially challenging aspect of ESD.)
• In assessing coverage of Sub-Targets the level of the whole dataset, we assigned an overall category score of 1 point per content area (for Sub-Target 1: Knowledge) or competency domain (for Sub-Target 2: Skills) that we judged to be partially covered, and 2 points per content area or competency domain that we judged to be systematically or rigorously covered. The definitions of partial coverage and systematic/rigorous coverage are provided in Table 2 and Table 3.

One complication occurred in the above process: it was found that one of the Skills sub-domains—namely L2K_CON (‘Understand sustainability concepts and issues’)—was so close to a description of Knowledge that was not possible to clearly allocate indicators to it. A distinction was implied between knowledge and understanding, but in practice the indicators did not provide such distinctions: for example, it seemed to attract most of the same qualifiers as for Knowledge. We thus excluded this sub-domain completely from analysis.

2.2.2. Consideration of Domain Overlaps and Non-Overlaps

A summary of domain differences between the IAEG-SDG and ESDinds indicator development approaches was provided in Table 1. The impact of those differences on the analysis for different components of the conceptual framework is illustrated by the shading shown in Table 2 and Table 3 above. Indicators arising from ESDinds were formed through the aggregation of more specific indicators, because part of the process of forming a transferable set across multiple organisations involved this. One consequence is that any natural mention of a specific knowledge area would have been subsumed into any parallel indicator relating to the practice of that knowledge—i.e., a related competency. Thus, with the coding rules as defined above, the ESDinds dataset is not expected to score in Sub-Target 1 (Knowledge) at all, as an artefact of its construction, meaning that the entire Sub-Target cannot be used for comparisons.

Similarly, the ESDinds approach was focussed on the present and near-future for the groups involved, and not the far future, and thus anticipatory competencies are not expected to be covered in its output indicators.

Both of these domains could easily be incorporated into future work using an ESDinds-type approach, with very little extra effort. Thus, they cannot be properly considered to be zero scores. However, in this retrospective study and with the specified scoring system, only zero scores are expected for Knowledge. The IAEG-SDG indicators were still processed, to allow consideration of its overall relevance of the approach to both Sub-Targets (Knowledge and Skills).

3. Results

In the sections that follow, we present overview summaries of the study findings; compare the item validity and sampling validity of the outputs of the two approaches where their domains overlap (i.e., Skills, other than Anticipatory Competencies); and compare sampling validity across each dataset as a whole at the third level of the conceptual framework, i.e., coverage of competency sub-domains. Specific examples from the coded text of indicators from both conventional and values-based approaches can be found in Appendix C, which readers would benefit from browsing. Full tables of coverage analysis results are presented in Appendix D, respectively illustrating the coverage of Sub-Target 1 (Knowledge) by the IAEG-SDG consultation responses (Figure D1); the coverage of Sub-Target 2 (Skills) by the IAEG-SDG consultation responses (Figure D2); and the coverage of Sub-Target 2 (Skills) by the ESDinds proto-indicators (Figure D3). The full coded datasets for IEAG-SDG and ESDinds are provided as Tables S1 and S2, respectively, in the Supplementary Material.

3.1. Overview of Relevance (Item Validity)

Table 4. An overview of the numbers of useful indicators for SDG Target 4.7 produced by the two indicator development approaches (conventional approach exemplified by Inter-Agency and Expert Group on Sustainable Development Goal Indicators (IAEG-SDG) public consultation, and values-based approach exemplified by ESDinds).

	IAEG-SDG Public Consultation Responses (Conventional Approach)	ESDinds Proto-Indicators (Values-Based Approach)
	Total N = 71 items	Total N = 125 items
Number of indicators with:
item validity score of 0 (not useful):	27 (38%)	56 (45%)
item validity score of 1 (useful for only ONE Sub-target):	30	69
Sub-Target 1 (knowledge acquisition only)	22	n/a
Sub-Target 2 (skills acquisition only)	8	69
item validity score of 2 (useful for both sub-targets)	14	n/a

provides an overview of the number of useful indicators for the operationalisation of Target 4.7, as produced by the two approaches. It is striking that a significant number of them were not useful for either the Knowledge or the Skills sub-targets. Given the specific request of the IAEG for responses to focus on Target 4.7, greater relevance might be expected. The second most striking point is that the UN-IAEG approach produced mostly knowledge-based indicators—but many of the respondents had expertise in ESD, and should have been aware of the skills aspects, so it is perhaps surprising that those were underrepresented. Thirdly, the ESDinds approach produced a large number of indicators generally (in the Skills category only—as predicted in Section 2.2.2). Given the original, local question they were designed to address, it is surprising that so many are relevant.

3.2. Overview of Coverage (Sampling Validity)

For Sub-Target 1 (Knowledge), the majority of relevant responses from the IAEG-SDG process covered only one or two content areas, with the most common ones being ‘gender equality’ and ‘appreciation of cultural diversity’ (see Figure D1 in Appendix D). It was notable that most of the candidate indicators from the IAEG-SDG process were concerned only with the acquisition of knowledge about sexual and reproductive health. These focused on measuring the outcomes of comprehensive sexuality education through two standard responses, proposed by UN agencies and repeated multiple times by other contributors. Two responses focused only on human rights. Some other responses mentioned knowledge in geoscience and environmental science—content areas mentioned in the target—but lacked specificity about what to assess. An exceptional contribution was provided by Brookings USA, which included detailed proposals relating to many knowledge domains (and skills, as discussed below). Only one response, from the United Nations Environment Programme (UNEP), called for universal application of the more holistic Sustainability Literacy Test [60] across the tertiary education sector. As anticipated, none of the ESDinds values-based proto-indicators mentioned knowledge acquisition at all. This was an expected artefact of the methods used to aggregate them into a set—see Section 2.2.2.

For Sub-Target 2 (Skills), there were far fewer relevant indicators from the IAEG-SDG process, and they generally had poor coverage of Target 4.7 (see Figure D2 in Appendix D). A number of UN consultation responses referred to the concept of “life skills based HIV [human immunodeficiency virus] and comprehensive sexuality education”, and some made a brief mention of, for example, vocational skills. Only a minority (n = 22, or 31%) included proposals for indicators that would actually measure the acquisition of skills relevant to content areas outlined in the target. In particular, it is striking that the coverage and number of indicators for the Sub-Target: Skills cluster of “Learning to Know” is extremely low (see Figure D2 in Appendix D), in contrast to the high coverage found in the Sub-Target: Knowledge (see Figure D1). This suggests a narrow view of the nature of knowledge, dominated by content rather than ways of knowing such as critical thinking or learning to be a self-driven learner and thinker. We explore this more fully in the Discussion section below. It is also interesting to note the unusually high number of hits for the sub-domain L2L_DIV within the competency domain of ‘learning to live together’, i.e., “Tolerate, respect, welcome, embrace, and even celebrate difference and diversity in people.

The results for the same Sub-Target 2 (Skills) from the ESDinds process had a very different profile (see Figure D3 in Appendix D). As expected, most ‘hit’ only one competency sub-category, as they were designed to be specific. However, as a set they produce very good coverage of Target 4.7, and in particular in the area that the IAEG-SDG process was weak i.e., in “Learning to Know”. Interestingly, like the other approach there is a large number of diversity-related indicators, but unlike it, there are also a large number relating to L2K_ACT (converting knowledge into action) and L2L_PAR (participation). It is also interesting that both approaches did not produce any indicators relating to the Anticipatory competencies category.

The overall coverage score for the full IAEG-SDG dataset was 7 out of a possible 16 points, representing partial coverage of all competency domains other than Anticipatory Competencies. For ESDinds, the overall coverage score was 9, representing systematic/rigorous coverage of Learning to Be and Learning to Transform Oneself and Society and partial coverage of all the other competency domains, with the exception of Anticipatory Competencies. The breakdown of results by sub-domain is shown in

Table 5. Number of indicators coded in each competency domain for candidate indicators from the IAEG-SDG public consultation (‘IAEG’) and proto-indicators from the ESDinds project (‘ESDinds’). Coverage is scored as 0 (none), 1 (partially covered), or 2 (fully or rigorously covered) for each domain.

	Learning to Do					Learning to Know				Learning to Be					Learning to Transform						Learning to Live Together					NOR	ANT	SYS	TOTAL SCORE
	L2D_ACT	L2D_APP	L2D_CRE	L2D_ISS	L2D_TRA	L2K_CRI	L2K_LEA	L2K_TAS	L2K_TOO	L2B_AUT	L2B_DEF	L2B_EXP	L2B_SEL	L2B_VAL	L2T_DEM	L2T_FOO	L2T_GEN	L2T_LIF	L2T_RES	L2T_SOL	L2L_CNF	L2L_DIV	L2L_ECN	L2L_PAR	L2L_UND
IAEG-SDG (full)	3	2	0	1	4	1	0	0	0	2	2	0	1	2	2	1	3	2	0	1	2	9	0	4	2	4	0	1
Coverage IAEG-SDG	1					1				1					1						1					1	0	1	7
ESDinds (full)	#	1	0	0	0	1	2	1	0	7	1	2	9	6	2	8	1	1	4	1	3	#	0	#	2	5	0	1
Coverage ESDinds	1					1				2					2						1					1	0	1	9

.

3.3. Sensitivity Analysis

Before proceeding to deeper discussion, we perform an analysis on the data to ensure that it is not inappropriately sensitive to small variations in the process by which each was created. To this end we removed the most significant contributor to each set of indicators. For the IAEG-SDG data it was noted that one particular respondent provided particularly useful and considered indicators—Brookings USA. These indicators achieved high coverage and relevance, and in that respect were anomalous to the other contributions. For a sensitivity test this single, most significant respondent’s contributions were removed, and the coverage obtained by the IAEG-SDG approach was re-calculated, as in

Table 6. Sensitivity analysis for coding of candidate indicators from the IAEG-SDG public consultation (‘IAEG’) and proto-indicators from the ESDinds project (‘ESDinds’). Coverage is scored as 0 (none), 1 (partially covered), or 2 (fully or rigorously covered) for each domain. In each case the most critical item (MCI) has been removed from the analysis. The figures for the full dataset are also included for the purpose of comparison.

	Learning to Do					Learning to Know				Learning to Be					Learning to Transform						Learning to Live Together					NOR	ANT	SYS	TOTAL SCORE
	L2D_ACT	L2D_APP	L2D_CRE	L2D_ISS	L2D_TRA	L2K_CRI	L2K_LEA	L2K_TAS	L2K_TOO	L2B_AUT	L2B_DEF	L2B_EXP	L2B_SEL	L2B_VAL	L2T_DEM	L2T_FOO	L2T_GEN	L2T_LIF	L2T_RES	L2T_SOL	L2L_CNF	L2L_DIV	L2L_ECN	L2L_PAR	L2L_UND
IAEG-SDG (full)	3	2	0	1	4	1	0	0	0	2	2	0	1	2	2	1	3	2	0	1	2	9	0	4	2	4	0	1
Coverage IAEG-SDG	1					1				1					1						1					1	0	1	7
IAEG-SDG (MCI removed)	2	1	0	0	4	0	0	0	0	1	1	0	0	2	2	1	2	2	0	0	1	6	0	2	0	3	0	0
Coverage IEAG-MCI	1					0				1					1						1					1	0	0	5
ESDinds (full)	#	1	0	0	0	1	2	1	0	7	1	2	9	6	2	8	1	1	4	1	3	#	0	#	2	5	0	1
Coverage ESDinds	1					1				2					2						1					1	0	1	9
ESDinds (MCI removed)	#	1	0	0	0	1	2	1	0	7	1	2	9	6	2	8	1	1	4	1	3	#	0	#	2	5	0	0
Coverage ESDinds-MCI	1					1				2					2						1					1	0	0	8

below. Similarly, the highest contributing indicator in the ESDinds set was removed. Table 6 shows that the IAEG-SDG score reduced from 7 to 5, where the highest possible score was 16. The ESDinds set performed better, reducing only from 9 to 8.

4. Discussion

4.1. Comparisons across Approaches

Figure 2 facilitates a broader comparison of the contribution of the two approaches to the coverage of Target 4.7 in the domain where they overlap—Skills. This in turn raises interesting questions about the intrinsic nature of the approaches and their potential use in future sustainability processes. Of particular interest is the low number of IAEG-SDG indicators in Learning to Know; the high absolute number of ESDinds indicators generally and in particular in Learning to Do and Learning to Be; and the fact that the highest relative number for both approaches is in Learning to Live Together.

4.2. Inside and Outside the (Traditional) ‘Box’

The questions posed by the UN’s SDG consultation and the ESDinds approach covered similar domains, but the UN consultation started with one proposed indicator and asked respondents to suggest alternatives—with the implication that any operationalising indicators returned would be appropriate to that given target, and that the responses would be in the context of traditional indicator types. The ESDinds project, despite having the same aim of generating indicators, began with the question of what practitioners find `meaningful, worthwhile and valuable’ in relation to their ESD work. This was usually done with a focus on successful projects that they had completed in the past, and the values underlying their success. Asking specifically for new candidate indicators may predispose respondents to thinking about indicators and data collection methods that are already familiar to them; but asking specifically about what is meaningful, worthwhile and valuable to people may help them to think outside the traditional indicator ‘box’. In this latter case, they may find it easier to identify less tangible dimensions of ESD, which usually fall outside the realm of evaluation and assessment [24]. The approach brought about useful results which could help to achieve the shift, noted by McCool and Stankey [39], from ‘what can be measured, using currently available methods’ to ‘what should be measured’. Figure 2 shows that ESDinds returned many indicators in Learning to Do, Learning to Be and Learning to Transform, which are not traditionally well covered by indicators.

As a way of portraying this, a phrase cloud is given below (Figure 3), which gives more of a feel for the nature of the indicators than the histogram in Figure 2. The smallest font indicates one indicator and the largest font represents 17 relevant indicators—’actor and thinker’ from ‘Learning_to_Do’ as generated by the ESDinds process. Less tangible areas such as ‘participate and cooperate’ and ‘self-actualise’ show up clearly.

4.3. Tacit Knowledge and the Values-Action Gap

The ESDinds indicators proved to cover many of the less tangible dimensions of ESD. One useful way of thinking about this result is that values elicitation, especially when based around reflection on collective projects that have been successfully completed, may provide access to tacit knowledge that has not previously been expressed in words. Tacit knowledge accumulates through shared contexts of practical experience [69,70] but may never be formally articulated unless—as in the ESDinds project and more recent adaptations of its methodology—it is actively elicited. The accumulation of tacit knowledge provides one explanation for a value-discourse gap—a situation in which people are already enacting a particular value in their day-to-day actions, but do not bring it into conversation [28]. A ‘value-discourse’ gap is the converse of the previously recognised ‘value-action gap’ [71], also known as an ‘environmental values/behaviour gap’ [72] or ‘attitude-action gap’ [73], in which people espouse a particular value in their discourse but fail to carry it forward into action [28]; see also [74,75].

The ESDinds approach has shown it can produce a large number of indicators covering areas which are generally considered to be much less tangible (see Appendix C for examples), and may make accessible tacit knowledge about the types of skills that are required for sustainable development. Such knowledge might not be foregrounded in traditional indicator development consultations, because of the existence of a value-discourse gap in the respondent organisations: aspects that are important in practice may not always be articulated in words.

4.4. Current SDG Discourse

The concept of value-discourse gaps takes on a greater significance if we regard the IAEG-SDG public consultation as a snapshot of contemporary global political discourse around the SDGs, and the Target 4.7 responses as an indication of how ESD is being discussed within the ‘SDGs community’. Remembering that the IAEG-SDG responses included 22 with relevance to Sub-Target 1 (Knowledge), it is noteworthy that only one was found to be relevant to the broader ‘ways of knowing’- such as critical thinking, or applying learning to life—described in the Skills, Learning to Know competencies. It seems that even though the target is about ESD, the SDG community is still focussing on the more familiar domains of the cognitive ‘learning to know’, e.g., in the sense of receiving knowledge, rather than challenging received knowledges or producing new systemic knowledges. Such traditional knowledge is characterised by an outward focus and readily measurable outcomes, e.g., those tested through examinations. The more affective and inward-facing dimensions of ‘learning to know’—as well as many other sustainability competencies like ‘learning to live together’, ‘learning to be’ and ‘learning to transform self and society’ and the ‘normative’ competencies that allow people to explore questions of values and ethics, and to critique prevalent definitions of SD itself—are frequently overlooked [28] (see also [21,22,25]). This is evident in the responses obtained through the IAEG-SDG approach, itself representing current discourse. This should not be taken to mean that such competencies are not valued, but rather that people’s individual tendencies to value them do not yet appear to have been translated into targeted conversations around how to facilitate their acquisition through education, and thus certainly not how to measure the success or failure of such efforts.

The ESDinds approach was designed to address this issue, and the indicators produced suggest its success. A case in point is L2B_SEL, “Develop one’s personality, self-identity, self-knowledge and self-fulfilment”, which had seven indicators proposed from ESDinds in comparison to the two from the IAEG-SDG approach. Thus, values-based approaches to indicator development may both support and promote an expansion of the global discourse on ESD to include these ‘inner’, ‘heart-centred’, or even `spiritually-oriented’ aspects that have eluded traditional approaches to indicator development. Such an expansion has been previously advocated in principle, although not demonstrated in terms of indicator development, by authors in both education [76,77,78] and sustainable design [79,80,81,82].

The absence of indicators relating to ‘learning to know’ and systems thinking in the IAEG-SDG public consultation is a matter of concern for two reasons: first, because the consultation can be seen as a reflection or indication of the current state of discourse, and second, because of its implications for the future. In terms of the state of discourse, it infers that meta-learning—i.e., having an awareness of the phenomenon of learning in general and of one’s own learning habits in particular, and taking control of one’s learning [83,84]—appears not to be a salient issue for the respondents. In a similar way, it suggests that systems thinking may not yet have been embedded in the way in which these respondents approach complex sustainability problems. In terms of implications for the future, the likelihood is that if an issue is not covered by the consultation responses, it is unlikely to be represented in the final indicator. Bearing in mind the old adage that “what gets measured gets done” [6], this would constitute a missed opportunity to embed a more radical shift in modes of education—from a transmissive to a transformative paradigm [48,78]—and to push schools, colleges and universities to produce graduates who are not only critical and creative `systems thinkers’, but also effective lifelong learners. This, in turn, has serious implications for the future of sustainable development, in that knowing about SD in a theoretical sense is very different from having the skills and motivation to take action, which in turn is not the same thing as actually taking action.

4.5. The Need for Localisable Indicators in ESD

Whichever indicator(s) the IAEG-SDG might ultimately select to operationalise Target 4.7, there will be a subsequent question of data collection that needs to be considered carefully. At one extreme, data for most indicators could be collected through a national-level survey, or even a simple ‘tick-box’ exercise within the respective government departments. This could easily become the default option, especially where resources are limited. At the other extreme, local institutions (such as teachers’ colleges, universities and schools) could take the indicator as a starting point for a variety of initiatives designed to report on the achievement of the SDG target within their day-to-day practice. These could include, for example, creating new assessment criteria and rubrics for assessing learning outcomes at the individual or group level [29]; auditing existing curricula, internal policies, organisational structures and codes of conduct in relation to the indicator; and/or evaluating the extent to which these internal policies are implemented, or guidelines adhered to, in practice.

The authors would argue that while it is important for the final indicator to be standardised to the point of comparability across institutions and countries, it must also allow for localisation in its application if it is to be perceived as relevant (a condition for engagement) and produce locally valid measures. In addition, local institutions should be strongly encouraged to develop their own initiatives to set up localised operationalisations, especially if the final indicator does not fully operationalise the skills dimension—as seems likely, on the basis of the responses. Without targeted action, there is a high risk that global conversations about ESD will end up focusing primarily on the transmissive model of education—focusing on acquiring particular kinds of knowledge [48]—and that the opportunity to embed transformative, competency-based approaches could be missed.

It may be helpful, in light of this discussion, to revisit Figure 1—the conceptual framework created in this paper. We propose this conceptual framework as a tool in its own right for use by local institutions (e.g., universities, schools, and civil society organisations) in their efforts to develop locally-valid ESD indicators, and apply them to the evaluation of organisational structures, policies, curricula, teaching practices and learning outcomes. The authors are currently developing new work in this area.

4.6. Salience and Emerging Breadth of Concepts

The existence of diverse candidate indicators or proto-indicators for a particular category, within a given context, may suggest that the category has been widely thought about and discussed within that group of people. In the ESDinds case, for example, we have noted that the category “L2D_ACT: Be an actor as well as a thinker” was used to code 17 different proto-indicators, in comparison to only three in the SDGs set. It is unsurprising but encouraging that this category is highly salient to the respondents, who are practitioners promoting ESD within the non-formal education sector—specifically in civil society organisations—within project contexts.

Other possible explanations for multiple indicators in a given category are that the category itself is difficult to differentiate from other categories within the framework; that it is too broad to be useful; or that the indicator development process is biased in favour of that category. In the case of L2D_ACT, for example, there is a significant overlap with two other categories whose wording inherently implies a shift from the realm of thought to the realm of action, namely “L2L_PAR: Participate and co-operate with others in increasingly pluralistic and multi-cultural societies” and “L2T_FOO: Promote behaviours and practices that minimise our ecological impact on the world around us”. However, even among those uniquely coded as L2D_ACT within ESDinds, there are several different manifestations. These include, for example, behaving in a way that is consistent with one’s words; actively helping the organisation to meet its goals; investing one’s own time and resources in activities that benefit the environment or society; and being motivated and productive in one’s work. This may suggest that there is a need for further conceptualisation of what is meant by “being an actor as well as a thinker”, which can most usefully be done at the local level. Such an observation reflects the point that the ESDinds items are proto-indicators: they serve as triggers or prompts for localised processes of reflection, conceptualisation and operationalisation, leading ultimately to indicators that are both salient and measurable at the local level.

The category with the highest overall number of indicators within the IAEG-SDG consultation dataset was “L2L_DIV: Tolerate, respect, welcome, embrace, and even celebrate difference and diversity in people”. In this case, the multiplicity of indicators is attributable largely to the wording of Target 4.7 itself, which explicitly references this competency: “Ensure that all learners acquire the knowledge and skills needed to promote SD, including, among others…appreciation of cultural diversity”. Thus, any indicators which replicated the wording of the target automatically scored a point for L2L_DIV. However, the same category was also associated with multiple indicators for ESDinds, where a number of different practical manifestations of welcoming, embracing and celebrating diversity were identified—e.g., accommodating different learning styles, giving people flexibility to do things within agreed boundaries, or providing different groups with equal access to information and decision-making. This highlights the possible need for further conceptualisation of L2L_DIV with reference to the literature on, and practice of, equality and diversity in both education and employment settings. As before, this may be most appropriately done at a local level.

4.7. Whose Targets? Whose Indicators?

Regardless of whether explicitly values-based approaches are used, our findings also open up a broader debate about indicator development in relation to the SDGs. Revisiting the United Nations call to “measure what we treasure” [5] in the light of this study, it becomes evident that what is not discussed is the meaning of ‘we’ in this statement. It appears to imply that a single set of values are being used as the benchmark for determining what is ‘treasured’, thereby setting a normative standard for indicator design—which seems to have gone unchallenged in this instance, but has previously been the subject of critique [39,85]. A more generous interpretation might suggest multiple definitions of ‘we’, and hence a nuanced and multi-subjective/intersubjective understanding of values and indicators—echoing the position of Ostrom [86] on the need for polycentric approaches, as well as Saith’s critique of the Millennium Development Goals (MDGs) [4]. However, neither the MDGs Report nor any subsequent document relating to measurement and assessment for the SDGs incorporates an explicit call for polycentric, decentralised approaches to SDG indicator development. Rather, as for the MDGs, the official process has been oriented towards identifying a single global indicator (or, if absolutely necessary, two or three indicators) for each of the 169 SDG targets.

Bell and Morse [12] highlight the ‘tyranny of methodology’ which is inherent in any attempt to reduce a multiplicity of stakeholder perspectives to a very limited set of global indicators. Whatever is ultimately produced by the IAEG-SDG as the indicator (or set of indicators) for Target 4.7 at a global level, we speculate that it is unlikely to satisfy any of its observers. Furthermore, we have illustrated in this paper that it is impossible for a single indicator or even a handful of indicators to provide valid measures of progress towards such a broad, multi-faceted, and intermeshed target, in the sense of covering the whole of the content domain [87,88].

4.8. Democratisation and Pluralism in Indicators

In highlighting the challenges of operationalising complex concepts such as ‘education for sustainable development’, we neither question the significance of the IAEG-SDG indicators and indicator design processes, nor the importance of global and national monitoring in a more general sense. Our intent is, rather, to suggest the relevance of supporting democratisation of the indicator development process. This would mean acknowledgement that designing indicators of progress towards complex goals like the SDGs is best served by a complementary processes involving both expert groups and international institutions and by local values and priorities at a variety of different levels.

The inductive development of indicators from what groups identify as immediately valuable, worthwhile and meaningful in their diverse contexts would have important implications for engagement in and the democratisation and diversification of global sustainability and development agendas as a whole. The inclusion of such indicator development processes in international conversations could draw increasing attention to diverse ways of seeing and understanding the world—leading, potentially, to the wider recognition of plural ‘dynamic sustainabilities’ in place of the IAEG’s single, static conception of sustainable development, implicit in the use of the definite article to refer to ‘the’ knowledge and ‘the’ skills required to promote SD. This is already an acknowledged issue for Indigenous communities, in the light of their long history of disenfranchisement and ongoing collective actions for international recognition. Indeed, the importance of creating SD indicators grounded in Indigenous values and world-views was formally noted in a 2006 technical report of the Permanent Forum on Indigenous Issues [17]. However, a trend towards democratisation and diversification could also open up the possibility of creating a place at the SDG table for other sustainability discourses that do not sit comfortably with the implicitly neoliberal concept of sustainable development [89]. These include, for example, the interface of the arts and sustainability [18]; work on ‘design for sustainability’ positioned in opposition to ‘design for sustainable development’, which demands a fundamental shift in values rather than a mere `technical fix’ of contemporary industrial societies [80,81,82,90]; Stephen Sterling’s positioning of ‘sustainable education’ in opposition to ‘ESD’ [77], grounded in similar concepts and arguments to those of ‘design for sustainability’; and the ecological philosophy of David Abrams, which posits that members of the ‘more-than-human’ community of life should be treated as perceiving subjects rather than as objects for human consumption [91].

Rather than advocating for the immediate dissemination and use of the specific proto-indicators created through the ESDinds project, our concluding action is to highlight the pragmatic usefulness of inductive values-based approaches in operationalising ESD targets at the project level, and—in light of this—to call for increased and wider engagement in national, regional and local initiatives generally, in order to complement the official IAEG-SDG process. In view of recognised difficulties in defining terms such as ‘global citizenship’ in a global way (and assuming that the same competencies will be equally important everywhere), we propose that the key point is perhaps the process of creating values-based indicators, as part of a global trend towards the democratisation of the SDGs.

5. Conclusions and Recommendations

A conceptual framework was constructed for SDG Target 4.7 and used to analyse the usefulness and character of indicators produced from a values-based approach compared to the UN approach. The findings suggest that the current SDG community, and within it those focusing on ESD, still conceive of or model knowledge in a narrow, traditional way relating to content—to the extent of not being able to suggest relevant indicators for different ways of knowing, such as critical thinking or `learning to learn’. Similarly, indicators for many of the less tangible areas of ESD which are not already easily covered by traditional indicator approaches were not produced by the ESD community involved in the UN’s IAEG-SDG process. Although this is likely to be a reflection of current discourse in that community, it is also likely to be a reflection of the IAEG’s consultation process, which appears to have difficulty generating non-traditional indicators. However, the consultation process did produce candidate indicators which partially covered most of Target 4.7.

The ESDinds approach produced complementary and occasionally better coverage of Target 4.7. It was specifically designed to produce indicators covering the less tangible areas of ESD, and the work presented here provides evidence that it did so, as well as opening up a discourse which included finely-developed concepts in several of the less tangible areas. Although the underlying premise in the ESDinds approach is to drive the process by considering what is locally valuable, worthwhile and meaningful, it is not clear to what extent to which its other aspects—e.g., its intersubjective and slightly disruptive elements, its purposeful contextualisation at group level, and its consideration of values in a holistic manner—contribute to those results. However, it seems likely that all three of these elements do contribute, because the resulting indicators produced included several representing group values in action, aspects not usually articulated, and aspects often considered intangible—which were not produced in the IAEG-SDG process.

Although it may initially seem unlikely that the ESDinds approach could contribute in practical ways to an indicator development program for complex targets, such as that of the IAEG-SDG, there are in fact some ways in which it could. Firstly, if the ESDinds-type approach were run by a small number of groups prior to wider consultation, it could produce candidate indicators (or even Targets) of those rare categories, which could then be used as prompts—both for the expert groups and wider public. Secondly, the IAEG-SDG process might be enhanced if contributors were advised to develop their candidate indicators through ‘disruptive’ or at least intersubjective, consultations at group level, rather than relying on single representatives of organisations. Lastly, it appears that even the ESD community which is cognizant of less traditional output types such as competencies (compared to narrow knowledge) did not propose related indicators, suggesting a hegemony of sorts which hinders untraditional measures or evidence. To get around this, the IAEG-SDG could encourage novel or non-standard ideas like those reflecting ‘heart’ or ‘culture’, or preferably produce examples of them for consultation. It certainly would need to try to include them when deciding on Targets and indicators, as they may be rare but they are valid—as made evident in our conceptual framework of Target 4.7.

Finally, the ESDinds type of approach of developing indicators from the shared values of a group, but articulating them in individualised action-related and context-related examples, may serve to relieve the tension between different indicator types. The ESDinds indicators are slightly generalised versions of specific examples that are important at the group level, and may be transferable to other groups, who can then localise them in their own context. The recommendation from this for future indicator development is to build in a ‘fuzzy framework’ of indicators, which allow localisation. For example, the indicator “People are not afraid to make mistakes” can be scored using evidence deemed locally to be relevant. Furthermore, whatever final indicator is constructed by the IAEG-SDG for Target 4.7, the authors would suggest that asking groups around the world to immediately consult and develop their own local interpretations and scoring systems as this would not only lead to more valid and useful indicators, but would also build awareness and capacity in ESD that produced a wealth of ideas to share with less creative groups. This could become an important part of the process of increasing ESD itself.

Lastly, the work presented here makes it clear that much more research is needed to explore the ESDinds process to better understand how it produces measures of that long-sought ‘intangible’: sustainability. Such work is likely to reveal new knowledge not only for sustainability, but also for other linked disciplines.