Next Article in Journal
From High School Access to Postsecondary Success: An Exploratory Study of the Impact of High-Rigor Coursework
Next Article in Special Issue
Systematizing Professional Knowledge of Medical Doctors and Teachers: Development of an Interdisciplinary Framework in the Context of Diagnostic Competences
Previous Article in Journal
Twenty-First Century Learning as a Radical Re-Thinking of Education in the Service of Life
Previous Article in Special Issue
Assessment of Competencies in Scientific Inquiry Through the Application of Rasch Measurement Techniques
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Education Sciences
Volume 8
Issue 4
10.3390/educsci8040190
education-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

How to Measure Procedural Knowledge for Solving Biodiversity and Climate Change Challenges

by
Lisa Richter-Beuschel
1,*,
Ingo Grass
2 and
Susanne Bögeholz
1
1
Department of Biology Education, University of Goettingen, Waldweg 26, 37073 Goettingen, Germany
2
Department of Agroecology, University of Goettingen, Grisebachstraße 6, 37077 Goettingen, Germany
*
Author to whom correspondence should be addressed.
Educ. Sci. 2018, 8(4), 190; https://doi.org/10.3390/educsci8040190
Submission received: 16 July 2018 / Revised: 25 September 2018 / Accepted: 27 October 2018 / Published: 30 October 2018
(This article belongs to the Special Issue Biology Education)
Browse Figures
Versions Notes

Abstract

:
To cope with biodiversity and climate change challenges, Education for Sustainable Development (ESD) needs to emphasize knowledge that considers multiple perspectives. Optimizing teacher education requires knowledge about the prerequisites of student teachers. The latter includes content knowledge with respect to Sustainable Development (SD). Apart from situational and conceptual knowledge, procedural knowledge (containing solution strategies) is of special interest, but it is much more difficult to measure. Thus, this study aims at developing a refined procedure to measure SD-relevant procedural knowledge and to define a measure for such knowledge, including a suitable benchmark for its evaluation. As SD-relevant knowledge, the SD challenges biodiversity loss and climate change were focused on. For operationalizing these challenges, the highly relevant contexts insects and pollination and peatland use were chosen. For both SD challenges and contexts, potential solution strategies were identified by a literature review. A procedure was then tested to measure procedural knowledge. The procedure includes a two-round expert survey (Delphi approach) with an in-between think-aloud study with student teachers. The described innovative procedure resulted in a measure (18 items) to assess procedural knowledge of student teachers via effectiveness estimations of provided solution strategies. This measure contains procedural knowledge items that are related to prior presented scenarios regarding the two contexts and a benchmark to evaluate these items. The benchmark derives from the second round of the Delphi study. The procedure and the developed final instrument include expertise from multiple disciplines such as ESD, SD, biodiversity, insect and pollination, climate change and peatland use. The sophisticated procedure can be transferred to challenging measurement developments. Furthermore, the measure provided for SD-relevant knowledge can be applied to other target groups in upper secondary and in higher education within ESD.

1. Introduction

Sustainable Development (SD) and sustainability are omnipresent concepts in today’s world. In nearly all programmatic documents on sustainable development, education is an important component [1]. It is internationally recognized that Education for Sustainable Development (ESD) is a substantial “element of quality education and a key enabler for sustainable development” [2]. ESD should serve to enhance the ability of children and adolescents to participate in establishing SD [1]. “ESD is expected both to make people more aware and better qualified to take part in shaping future developments responsibly and to raise their awareness of the problems related to sustainable development and bring forth innovative contributions to all economic, social, environmental and cultural issues” [3]. In the years 2005 to 2014, the United Nations (UN) Decade of ESD fostered the role of education regarding SD [2]. Also, one of the 17 Sustainable Development Goals (SDGs) addresses education. Goal Four reads, “Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all” [4]. Particularly, the associated “Target 4.7” deals with SD. In the Global Action Programme (GAP) on ESD of UNESCO (2015–2019), the “building capacities of educators and trainers” is one of five priority action areas identified [2]. Therefore, the education of future multipliers is an objective to support SD in a wider range. Because of its potential multiplier effect [5,6], teacher education has a key function in fostering ESD.
Target 4.7: “By 2030, ensure that all learners acquire the knowledge and skills needed to promote sustainable development, including, among others, through education for sustainable development and sustainable lifestyles, human rights, gender equality, promotion of a culture of peace and non-violence, global citizenship and appreciation of cultural diversity and of culture’s contribution to sustainable development”.
[4]
Despite the UN Decade, teacher training has been insufficient so far. “To date there are hardly any structural changes in teacher education and training, and there is a need for development in the school structures and curricular requirements in order to promote the ESD concept” [6]. There are several initiatives to promote ESD in teacher education. There are joint networks for teacher education promoting ESD around the world (e.g., the United States Teacher Education for Sustainable Development Network [7], the Teacher Education for Equity and Sustainability Network (TEESnet) in the UK, and the German­speaking network Teacher Education for a Sustainable Development (LeNa) [8]).

1.1. Teacher Education and Knowledge Relevant for ESD

Regarding teacher education, professional action competence is crucial for successful teaching [9,10] and can decisively impact ESD [11]. Professional action competence is composed of professional knowledge along with motivational, volitional, and social willingness and skills [12]. Professional knowledge can be divided into content knowledge (CK), pedagogical content knowledge (PCK), and pedagogical knowledge (PK) (cf. [9,13]). Based on Shulman [13], Baumert and Kunter [9] developed a competence model for mathematics, which was adapted for ESD by Hellberg-Rode, Schrüfer and Hemmer (cf. [14,15]). Cognitive competencies, which teachers are supposed to be equipped with, regarding ESD could be extracted in an expert study. Thirty-eight percentage points of these competencies were assigned to CK, 23% to PCK, and 19% to PK. Therefore, CK plays a critical role regarding ESD and should be focused on in this contribution.
To optimize ESD in teacher education, knowledge about the prerequisites for teachers facing SD challenges is essential [16]. Kaiser and Fuhrer [17] complain about the often undifferentiated measurement of knowledge which may result in underestimating the role of knowledge (cf. [17,18]). In the literature, there exists a variety of knowledge classifications (e.g., [19,20]). A common distinction is made between know-that/what (declarative, conceptual knowledge) and know-how (procedural, strategic knowledge) [21,22,23]. Challenges of SD are often situations of high complexity with conflicts between different legitimate objectives, e.g., ecological, economic, social, and/or institutional perspectives. Therefore, solving problems is crucial for coping with SD challenges. The knowledge model of de Jong and Ferguson-Hessler [23] explicitly focuses on problem-solving and thus on knowledge-in-use. The model defines four types of knowledge: situational, conceptual, procedural, and strategic knowledge [23]. Situational knowledge comprises the knowledge about domain-specific situations. For example, this knowledge allows extracting relevant information from a given problem description and adding further information [23]. Conceptual knowledge contains “concepts, facts and principles that apply within a certain domain” [23]. It allows the problem solver to change the state of the problem. Procedural knowledge comprises actions that are suitable to certain types of problems in the specific domain [23]. Strategic knowledge comprises the sequential action steps which are necessary to solve a problem and is “applicable to a wider variety of types of problems within a domain” [23]. Other authors often do not distinguish between procedural and strategic knowledge, but rather include the step-by-step actions for problem-solving in the term procedural knowledge [19,24,25,26].
The model of de Jong and Ferguson-Hessler [23] turned out to be well applicable for measuring knowledge in the domain of SD. For example, the model was picked up to determine the knowledge of Indonesian university students concerning resource use problems [23,27]. With the developed measurement instrument, situational, conceptual, and procedural knowledge (according to de Jong and Fergusson-Hessler [23]) were recorded (n = 882) [27]. Situational and conceptual knowledge were measured by multiple-choice items. The single items focused mainly on ecological, socio-economic or institutional questions [27]. For answering questions on situational knowledge, information from problem descriptions, presented as scenarios, had to be extracted and complemented. For answering questions on conceptual knowledge, knowledge about concepts and facts had to be applied.
In contrast to situational and conceptual knowledge, the different nature of procedural knowledge in the ESD domain requires one to consider core features of the concept of SD. Instead of clear (more discipline-focused) solutions like for situational and conceptual knowledge, different perspectives such as economic, ecological, institutional, and social factors have to be taken into account simultaneously [28] when evaluating a solution strategy. Procedural knowledge is described as “the cognitive skill of identifying and judging potential solutions (‘strategies‘)” [27] of environmental problems.
For being able to evaluate procedural knowledge of students in higher education, Koch et al. [27] conducted an expert study (n = 9) to establish a benchmark. The expert benchmark allowed them to judge the student answers concerning procedural knowledge items. The benchmark resulted from a single expert survey. The procedural knowledge items of the benchmark were developed by the authors without integrating further considerations and suggestions on problem-solving strategies of the experts. Thus, the procedure for benchmark development can be improved with respect to more intensively incorporating the expertise of the participants. Koch et al. [27] showed how to measure knowledge for typical Indonesian resource use dilemmas, such as rattan extraction from the rainforest or dynamite fishing, in higher education in Indonesia. The study demonstrated substantial gaps in the students’ ability to solve complex environmental problems [27].
For European teacher education and even teacher education in other continents, the Indonesia-related resource use dilemmas are not of similar relevance as they are for education in Indonesia. Therefore, this contribution focuses on challenges of SD relevant on a global scale: biodiversity loss and climate change [29]. Both are implied in the ESD goals [29]. Several studies reported that science teachers show large deficits in their biodiversity-related knowledge [16,30,31,32,33]. For example, biology student teachers are unaware of the core facets attached to biodiversity, such as its sustainable use [33]. Furthermore, student teachers’ understanding of “the terminology, distribution, and loss of biodiversity” [16] does not equate with scientific understanding. Studies with teachers, pre-service teachers, and student teachers also revealed gaps in knowledge and understanding, e.g., misconceptions of the greenhouse effect and (the causes of) global warming [32,34,35,36]. However, knowing the concepts of biodiversity and climate change, with the complex relations between ecological, social, and economic factors, is crucial for ESD. Thus, the ability to change perspectives is an indispensable ESD-specific competence [15,37,38,39] and highly important for procedural knowledge in the domain of SD [27]. However, interdisciplinary approaches to face controversial SD challenges are hardly implemented in science courses (cf. [40]).

1.2. SD Challenges Biodiversity Loss and Climate Change and the Corresponding Fields of Action

For the SD challenges biodiversity loss and climate change, there are many possible contexts. In the following section, an up-to-date, highly socially relevant and exemplary context concerning local and global biodiversity is described: insects and pollination. Second, a context with noteworthy impact on climate change is exemplified: use of peatlands.
Numerous crops and wild plants depend on insects as pollinators, particularly bees. Besides domesticated pollinators (honeybees), wild pollinator populations play an important role [41]. Seventy-five percent of leading global food crops rely on animal pollination. In total, 35% of global production depends on pollinators [42]. The global economic value of insect pollination amounts to about €153 billion per year [43]. In the last few decades, managed honeybee populations, as well as wild pollinators and plants that depend on their pollination services, declined [41,44]. A combination of different factors may be the cause of these declines: land use intensification with loss and fragmentation of habitats in agricultural landscapes [45,46], improper use of plant protection products, particularly neonicotinoids [44,47,48,49], and diseases or parasites like the Varroa mite [50].
In Europe, over the last century, 52% of total peatland area has been converted for agriculture, forestry, and peat extraction [51]. Peatlands influence the global climate system, particularly through carbon dioxide (CO2) and methane (CH4), and less through nitrous oxide (N2O) [52]. Because they accumulate more carbon through photosynthesis than they release through respiration, most uncultivated peatlands are sinks for atmospheric CO2 [51]. Simultaneously, most peatlands emit the powerful greenhouse gas methane (CH4), caused by the wet conditions [51]. The most accumulation of peat occurred within the past 10,000–20,000 years [52]. Land use activities like agriculture and peat extraction for potting soil require drainage of the peatland. Consequences of drainage include enhanced decomposition of peat, the generation of CO2 and N2O emissions and reduced CH4 emissions [52].
Thus, insects and pollination and peatland use are major contexts within the SD challenges biodiversity loss and climate change. It is common practice in current research, e.g., on knowledge [27] or on socio-scientific reasoning [53], to incorporate two socio-scientific issues (rattan extraction and dynamite fishing in [27], land use management issue and fracking in [53]) to assess the respective competencies. To measure the constructs to be investigated, independent from prior knowledge, problem descriptions are provided in the form of scenarios. This is conventional in competence research, e.g., in Programme for International Student Assessment (PISA) [54].
Recently, measures for situational and conceptual SD-related knowledge have been developed for the following contexts: insects and pollination and peatland use [55,56,57]. To complete the SD knowledge measurement instrument in the domain of SD, a measure for crucial procedural knowledge is still lacking. The term “measure” is used for clearly distinguishing from the term “measurement instrument”, which additionally includes situational and conceptual knowledge. Therefore, the core research question is: how can interdisciplinary SD-related procedural knowledge of student teachers be measured? The overall study aims to:
(i)
refine a procedure to measure procedural knowledge of student teachers for coping with SD challenges and thereby,
(ii)
define a measure for such knowledge and a benchmark for its evaluation.

2. Methods

In the following section, the methodological approaches for developing a procedure to measure SD-relevant procedural knowledge of student teachers will be presented. In the current project, procedural knowledge refers to the domain of SD. Procedural knowledge equates with the cognitive skill to solve certain SD-related problems (cf. [23]). Therefore, considering and weighing different perspectives is necessary.
The development of a measurement instrument for SD-related knowledge of student teachers considers the contexts of insects and pollination and peatland use. Both are complex environmental problems requiring an interdisciplinary and/or multidisciplinary perspective. According to common practices [27,53], two scenarios of the real-world problem (one for each context) are given. They form the basis for evaluating situational, conceptual and, presented in this study, procedural knowledge. One scenario deals with a large bee colony loss in Germany in 2008 [49]. The second context deals with the agricultural and industrial use of peatlands. The scenarios (abbreviated and slightly modified) are printed in Appendix A and Appendix B.
With respect to developing the measure for procedural knowledge, the following steps have been conducted: first, a literature review was performed to gather proposed solution strategies for SD challenges. Second, a two-round Delphi study was realized (cf. [58]). The Delphi procedure was enriched by an intermediate think-aloud study with student teachers (Figure 1). Before answering the questionnaire regarding solution strategies for both contexts, the students received the above-mentioned corresponding scenarios. The latter was not given to the experts. It was assumed that experts possessed such knowledge. The whole procedure serves to develop sets of potential solution strategies that are literature based, reviewed by experts and accessible to the understanding of student teachers. The elaborate procedure will result in a measure for a survey on SD-relevant procedural knowledge of student teachers (Figure 1).
According to common practice (cf. [27,53]) two contexts were chosen to operationalize procedural knowledge: insects and pollination and peatland use. The solution strategies proposed in the items require interdisciplinary considerations that integrate environmental, socio-economic, and institutional issues. To broadly cover both exemplified SD challenges, items were deduced from eight overarching topics: environmental policy, ensuring the diversity of species, sustainable management, and ESD (see Section 3) [58].
During the process of questionnaire development (Figure 1), a special questionnaire format was used. In the Delphi survey, the experts were asked to rate the effectiveness of given solution strategies on a four-point Likert scale, from “ineffective” to “very effective”. Each solution strategy was rated for three essential fields of action. For the insect and pollination context, the solution strategies were estimated concerning: (i) realization of sustainable land use, (ii) provision of ecosystem services, and (iii) biodiversity conservation. For the peatland use context, the solution strategies were equally reviewed regarding (i) and (ii), whereas (iii) consisted of the contribution to climate protection (Figure 2). Each expert had to rate effectiveness in both contexts. In addition to the three answers per solution strategy, the experts gave information about how certain they were about their effectiveness estimations on a four-point Likert scale, from “absolutely uncertain” to “very certain” (Figure 2). The evaluation of subjective certainty of estimations is a characteristic of Delphi surveys [59,60].
In addition to the procedural knowledge assessment, the experts were asked to self-assess their knowledge regarding eight topics: biodiversity, bees and pollination, climate change, the importance of peatlands, sustainable development, sustainable land use, environmental policy, and ecosystem services. The experts could choose “unsatisfactory”, “satisfactory”, “good”, “very good”, or “excellent”. Each expert answered the questions once, even if they participated in both rounds. The entire procedure was conducted in German.

2.1. Sample Composition

Experts from different disciplines were invited to participate in the Delphi survey. Potential participants had expertise as scientists in subjects such as teacher education, biology, climatology, or agricultural sciences. The final participant list included 15 professors, four postdoctoral scientists and four persons with unknown academic degrees. They came from nine German universities as well as from four non-university institutions (Table 1). All participants were people who deal with ESD, biodiversity, climate change or real-world problems of insects and pollination and peatland use in a wide range of fields.
For the first Delphi round, 27 experts were invited to participate. Among them, 19 answered the questionnaire in time (five female, 13 male, one not stated). The average age of the experts was 46.6 years (standard deviation (SD): 9.3). In the second Delphi round, 30 experts were invited, and 21 answered the questionnaire in the given deadline (five female, 15 male, one not stated). One person was subsequently excluded because of a lack of expertise (youngest scientific age and high self-assessment in contrast to the other experts). One person of the remaining experts only answered the peatland use context items. The average age of the 20 experts was 50.4 years (SD: 8.7). Three persons were between 31 and 40 years of age, nine were between 41 and 50 years, five were between 51 and 60 years, and three were older than 60 years. Sixteen of the 20 persons also participated in the first Delphi round. Table 1 displays the sample compositions with all participants of the first and second Delphi round. All experts of both Delphi rounds participated without any expense allowance.
The think-aloud study was conducted with nine student teachers: one bachelors and two masters students in biology, two bachelors and one masters student in geography, and two bachelors students and one student after his studies in political science.

2.2. Delphi Survey First Round

Based on the literature review, 41 solution strategies (Likert scale items) for insects and pollination (21 items) and for peatland use (20 items) were identified [58]. Apart from estimating effectiveness in the three essential fields of action and the certainty of the effectiveness, the experts had the opportunity to comment on the solution strategies and make suggestions for further solution strategies in an open-ended format. The qualitative data generated by comments and suggestions were analyzed through qualitative content analysis according to Mayring [58,61]. This analysis resulted in clarifying and optimizing items and determining further new solution strategies. Two items concerning agricultural subsidies originated from suggestions made by the experts in the first Delphi round (IP-8 and PU-10, see Section 3).
The processing time of the questionnaire with all of its elements (estimations of effectiveness, estimation of certainty, comments, suggestions for solution strategies, and self-assessed knowledge) amounted to a maximum of 45 minutes. For the quantitative analysis, Excel (Microsoft Office Professional Plus 2013) was used. Statistical measures like means, medians and standard deviations were calculated with the quantitative data. Medians and percentage distribution within the response categories were used for providing graphically documented results to the participants of the second Delphi round (Figure 3). In some cases, two boxes were marked for one rating scale. Instead of excluding these answers, it was assumed that the experts wished to have a scale with higher resolution. Therefore, the neighboring values were averaged.
Taking into account: (i) the results of analyzing the items from the questionnaire draft; (ii) the item difficulties; (iii) the comments and suggestions from the experts; and (iv) the representation of the eight overarching topics, a first revised version of the questionnaire with 27 items was created (Figure 1). These 27 items cover 14 items in the insects and pollination context and 13 items in the peatland use context.

2.3. Think-aloud Study with Student Teachers

To develop a procedural knowledge test for teacher education purposes, the procedure requires testing with (prospective) teachers. Therefore, a think-aloud study was performed with nine student teachers (Figure 1; for the method, see [62]) with the first revised version of the questionnaire (27 solution strategies; Figure 1). The think-aloud method is well suited to checking how items are perceived by subjects and a common method of item validation [63]. The think-aloud protocols were transcribed and analyzed through qualitative content analysis according to Mayring [61,64].
Like the experts, the students assessed the effectiveness of solution strategies on a four-point Likert scale in three fields of action: (i) realization of sustainable land use, (ii) provision of ecosystem services, and (iii) biodiversity conservation in the insect and pollination context or contribution to climate protection in the peatland use context (Figure 2). In contrast to the experts, the students received background information on two scenarios of real-world problems before answering the questionnaire (Appendix A and Appendix B). The objective was to establish a homogeneous knowledge base on which procedural knowledge could be assessed.
The aim of the think-aloud study was to adapt the questionnaire material for student teachers. Thus, problems concerning the two scenarios, the corresponding solution strategies (items), and the three fields of action were identified. Problems in understanding items led to rewording of those items, e.g., several items were linguistically simplified. Sometimes, a supplement in the sentence was added to make the strategy presented more understandable for student teachers. For example, in the solution strategy “The government provides financial incentives for using bee-friendly bloomers as biomass in biogas power stations”, the phrase “as biomass” was appended after the think-aloud study. Several students were confused with the former formulation, e.g., Sandra mentioned, “I do not know if biogas power plants [...] are the areas where plants are grown that produce biogas? I have no idea at all” (134–136, translated from German). In addition to item revision and optimization, the think-aloud study provided indications for the potential removal of items, as we were aiming for a short measure of 20 items broadly covering both contexts. Furthermore, the think-aloud study revealed a need to explain the meanings of “sustainable land use”, “ecosystem services”, and “biodiversity” for student teachers. This resulted in editing a supplementary informational sheet containing brief definitions of these terms.

2.4. Delphi Survey Second Round

Controlled feedback and statistical aggregation of all participating experts from the first round responses are key characteristics of a Delphi survey [65]. Therefore, in the second Delphi round, the experts received the graphically processed results of the first Delphi round, the supplementary information sheet containing the requested definitions of fields of action, and the second revised and condensed version of the questionnaire (20 items, Figure 1). Ten items addressed solution strategies for insects and pollination and 10 items addressed solution strategies for peatland use. Again, the experts had to assess these strategies regarding their effectiveness concerning the three fields of action and indicate their certainty of their effectiveness assessment, as in the first round (Figure 2). The processing time of the questionnaire took 15 to 20 minutes. The latter was due to the reduced item number, no further requested comments or suggestions and the self-assessment of knowledge was only necessary for those who did not participate in the first round.
The processing of the quantitative data of the second Delphi round finally aimed at establishing a benchmark for being able to evaluate procedural knowledge of student teachers. For the quantitative analysis, IBM SPSS Statistics 24 and Excel (Microsoft Office Professional Plus 2013) were used. The four-point Likert scales for effectiveness were coded from 1 (ineffective) to 4 (very effective). Like in the first Delphi round, if two boxes were marked, the neighboring values were averaged. Because of the two different contexts and due to differing expertise, the experts’ answers were weighted with the given certainty [66] for more reliable results. Therefore, the function “weight cases” in SPSS was used. With the weighted values, means and standard deviations were calculated. These statistics were calculated separately into the three fields of action. Additionally, these measures were calculated for each solution strategy across the three fields of action. We call this the “total weighted effectivity with respect to SD challenge”, or in short version “SD effectivity”.
For each of the three fields of action, a reliability analysis was performed with the expert data on effectiveness estimations. Because of a lack of procedure in SPSS to weigh the items with different variables, the unweighted values were used for the analyses. In addition, a reliability analysis with weighting over averaged certainty estimation was performed. Cronbach’s alpha was computed across the 10 solution strategies for insects and pollination and peatland use and separated according to the three fields of action. Furthermore, analyses of significant differences between the fields of action of each solution strategy were tested by repeated measures ANOVA (rmANOVA) with unweighted values. To check the validity of the expert data, the experts’ self-assessment of knowledge was correlated with their averaged certainty for each context. Due to the data being not normally distributed, Spearman’s rho (rS) was used for correlation analysis.

3. Results

Considering the quantitative data of the second Delphi round is essential for generating a benchmark for assessing procedural knowledge. Looking at the SD effectivity of the insects and pollination context, effectiveness corrected by certainty estimations (weighted) ranges from 2.21 to 3.71 (Table 2). The range of weighted effectiveness for the 10 items of peatland use is from 2.54 to 3.50 (Table 3). The mean SD effectivity across all solution strategies of each context amounts to 3.01 for insects and pollination and 3.01 for peatland use, too. The standard deviation in the insects and pollination context is about 0.10 lower (0.64) than in the peatland use context (0.74). The means and standard deviations of the individual fields of action are shown in Table 2 and Table 3. The solution strategies are presented in ascending order with respect to weighted effectiveness.
In contrast to the presented results of weighted effectivity estimations, the following data analyses are conducted with unweighted effectivity estimations (see Section 2.4)—if not indicated otherwise. In the following table, differences between the means of the single fields of action of sustainable land use, ecosystem services, and biodiversity conservation or climate protection are presented. The repeated measures ANOVA (partially with a Greenhouse–Geisser correction because of a lack of sphericity) reveals five statistically significant differences in solution strategies for the insects and pollination context and two for peatland use (Table 4). The respective effect sizes (partial η²) indicate large effects (Table 4; ≥0.01–<0.06: small, ≥0.06–<0.14: moderate, ≥0.14: large, cf. [67]). Also, four solution strategies show tendencies (≤0.10) (Table 4). The Bonferroni-adjusted post hoc analysis revealed eight statistically significant differences between the fields of action (marked in Figure 4 and Figure 5, cf. Appendix C).
For the insects and pollination context, the item “Align the breeding of honeybees for resistance to disease and parasites” (IP-1) is rated to be more effective for ecosystem services than for the realization of sustainable land use and biodiversity conservation (Figure 4). The item “Realign agricultural subsidies to stop promoting conventional and intensive agriculture” (IM-8) is rated to be more effective concerning sustainable land use than the provision of ecosystem services and biodiversity conservation. The item “Design a cultural landscape to serve pollinators as food source and habitat” (IP-9) is rated to be more effective in biodiversity conservation than in ecosystem services. In contrast, the item “Strengthen the protection of wild bees and other pollinating insects” (IP-10) does not differ between biodiversity conservation and ecosystem services. However, IP-10 is rated more effective for biodiversity conservation than for sustainable land use, which does not account for IP-9 (Figure 4).
For the peatland use context, the items “Cultivate peatlands without fertilizers and pesticides” (PU-6) and “Apply existing laws stricter, e.g., prohibit the converting of grassland into maize cultivation” (PU-9) are rated to be more effective in sustainable land use than in climate protection (Figure 5).
In short, the results in the essential fields of action of sustainable land use, ecosystem services, and biodiversity conservation or climate protection differ partly. Thus, the individual consideration provides more information than the SD effectivity.
Reliabilities were calculated with unweighted effectivity estimations for consistency of data analysis. Cronbach’s α values for the insects and pollination context indicate satisfactory (biodiversity conservation = 0.758) to high reliabilities regarding fields of action (sustainable land use = 0.838, and ecosystem services = 0.887) (Table 5). In the peatland use context, the reliability of the fields of action ecosystem services and climate protection is acceptable (0.763 and 0.764, respectively); the reliability of sustainable land use is relatively low (0.572). The reliability over both contexts in sustainable land use (0.852) and ecosystem services (0.906) indicate reliable scales (cf. [68]). For the reliability analyses, a weighting with the averaged certainty of the 10 solution strategies per context is a feasible procedure. The results differ by less than 0.01 from the reported values—except for the peatland use context (see Cronbach’s α values in brackets in Table 5 with, e.g., 0.633 instead of 0.572 for sustainable land use).
For validation purposes, the certainty of estimations in the two contexts from the experts was correlated with their fitting self-assessed knowledge. Therefore, special variables were computed. Regarding the insects and pollination context, the mean of a variable combining self-assessed knowledge on biodiversity, bees and pollination, and sustainable development was calculated and correlated with the averaged certainty about all 10 solution strategies. The correlation is large and statistically significant (rS = 0.73, p < 0.001; r > 0.5: large (cf. [68])). A large effect and significance could also be identified for the correlation of averaged certainty about all 10 solution strategy items from the peatland use context, with the mean of a variable consisting of self-assessment concerning the importance of peatlands, climate change, and sustainable development (rS = 0.53, p = 0.016). Thus, participants who assessed their knowledge as lower seemed to be less confident in answering the corresponding solution strategies. The results underline the validity of the expert certainty estimations.

4. Discussion

Here we present a refined procedure to measure procedural knowledge with a multi-level Delphi approach. The two-round Delphi survey with an intermediate think-aloud study enabled us to develop a measure for procedural knowledge including a benchmark to evaluate such knowledge of student teachers in the future.
The procedure consisted of using solution strategies for SD challenges according to the literature [58], estimations of such solution strategies and further suggestions from experts in the first Delphi round and item revisions due to the results of a think-aloud study with student teachers. The resulting measure contains 20 items reflecting the SD challenges of biodiversity loss and climate change. In the second Delphi round, the 20 solution strategies were estimated concerning effectivity with respect to three fields of action. Furthermore, the certainty of three effectivity estimations per solution strategy item was judged by the experts.
Regarding the sample composition of the Delphi study, experts from a broad range of expertise throughout Germany participated (Table 1). In addition, non-university experts contributed (e.g., from the Ministry of Environment and from a research institute). The fields of expertise cover multiple perspectives with respect to ESD, SD, biodiversity, insect and pollination, climate change, and peatland use. Furthermore, the degree of expertise was very high, comprising 15 professors among the 20 participants of second Delphi round.
In contrast to an expert study, the two-round Delphi study allowed us, for example, to integrate new solutions strategies in the second Delphi round that were suggested by the first round participants. In the present study, the solution strategies “Realign agricultural subsidies to stop promoting conventional and intensive agriculture” (IP-8, see Table 2) and “Provide agricultural subsidies only for sustainably managed peatlands” (PU-10, see Table 3) were newly proposed in addition to the 41 solution strategies provided by the authors in the first Delphi round [58]. These two solution strategies were integral components in the second Delphi round questionnaire. Furthermore, the Delphi procedure chosen includes that the participants in the second round can work with the expert knowledge from the first Delphi round, e.g., the graphically processed results of the questionnaire, judging upon revised solution strategies. In sum, the two-round Delphi procedure enriches the knowledge base for the assessment of—in the present study—SD-relevant interdisciplinary knowledge.
Strengths of the performed Delphi study are: (i) the great number of participants, (ii) the high degree of (scientific) expertise of the participants, (iii) the repeated questioning of the experts, and the possibility of the experts to suggest further solution strategies in contrast to other expert studies aiming at establishing benchmarks for assessing procedural knowledge [27,69]. Another strength of the Delphi study presented is the diversity of the disciplines of the participants involved, which enriches the input given with respect to SD challenges. The expertise behind the measure of procedural knowledge for student teachers proposed in Table 2 and Table 3, as well as the respective benchmark information for evaluating such knowledge, is deeply integrated in the measure. The presented measure resulted from a two-round Delphi study instead of a single data collection.
Another aspect of the Delphi study is the weighting of the experts’ answers. Despite being legitimated through the use of two different contexts [66], the weighting can lead to a bias, because potentially self-critical persons chose a moderate confidence level despite high knowledge while self-confident persons chose a high confidence level [70]. However, tests have shown that subjective certainty builds an indicator for the quality of estimations [59]. An alternative method could be to only consider experts with high self-assessments [66]. The latter would lead to different sample sizes in the two contexts because the self-assessment concerning biodiversity and bees is lower than that concerning climate change and peatlands. However, in the present study, weighting the estimations through certainty seems to be the best method for considering the individual knowledge of the experts. Besides, one participant with evidently biased responses was excluded from the analyses of the second Delphi round (see Section 2.1). Applying the method of using weighted effectiveness estimations of solution strategies through considering certainty estimations is also supported by the fact that experts with lower self-assessed knowledge seem to be less confident in answering the solution strategies.
To determine the benchmark, it is recommended to exclude one item for each context. For the insect context, the item “The legislator prohibits application of neonicotinoids” (IP-6) should be excluded because, on 27 April 2018, the EU Commission prohibited the use of neonicotinoids based on the substances clothianidin, thiamethoxam, and imidacloprid. On the same day, the invitation to participate in the second Delphi round was sent to the experts. Because most abundant neonicotinoids are prohibited, this is no longer possible but rather an implemented strategy and therefore negligible. Solution strategy IP-6 was one of three items of the overarching topic “environmental policy”. Accordingly, this topic will still be represented.
Also, the item “Individuals purchase products only from sustainable peat extraction” (PU-2) should be excluded from the prospective benchmark. Contrasting all other items, solution strategy PU-2 does not derive from the literature review. The original objective was to create an ineffective item. Because of the very slow growth rate of peat, despite ideal conditions, sustainable extraction of peat is virtually impossible. The exclusion of PU-2 no longer represents the overarching topic of “sustainable consumption”. Contrasting, the relevance of sustainable consumption in the insects and pollination context, this topic could be neglected in the peatland use context, as the peat consumption is not as widespread as the use of agricultural products requiring pollination services. By eliminating items IP-6 and PU-2 from the benchmark, all remaining 18 items are valid regarding the actual legislation as well as the literature base. In addition, they cover a wide range of essential overarching SD-relevant topics. By extracting the two items, the reliability of expert ratings only slightly differ—except for sustainable land use in the peatland use context (Table 4). All in all, for the measure of procedural knowledge, the reliabilities of the prospective data of student teachers is important.
As a result of the project, a final item set for a measure for interdisciplinary procedural knowledge concerning CK for ESD is provided. It includes two contexts concerning SD challenges. The insect and pollination and the peatland use contexts cover 18 items in total (Table 2 and Table 3, without IP-6 and PU-2) for a differentiated assessment of procedural knowledge in three essential fields of action: the realization of sustainable land use, provision of ecosystem services, and biodiversity conservation or climate protection. These fields of action are all interdisciplinary and/or multidisciplinary. Within the fields of action, the items either have clear foci (e.g., PU-8: “Intensify the investigation of regenerative peat substitutes”) or cover broader ESD requirements beyond the assessed fields of action (e.g., IP-10: “Strengthen the protection of wild bees and other pollinating insects”). Thus, the two-round Delphi study, complemented by an intermediate think-aloud study with student teachers, turned out to be a suitable procedure for generating an instrument with an included benchmark for measuring SD-related procedural knowledge of student teachers.
The developed measure as well as the measurement instrument can likewise be applied to other target groups, such as students in higher education in academic fields such as biology, geography, and sustainability, as well as in secondary school education for students. The chosen contexts of insects and pollination and peatland use are suitable to European countries and even other continents. When students receive the questionnaire (without item IP-6 and PU-2), different evaluation procedures of the benchmark are possible: to correlate the answer profile for each student teacher with the profile of the experts (cf. [27]) or to compare the ranking of effectiveness between experts and student teachers (cf. [69]). Furthermore, the innovative procedure used to develop the measure can be applied to cope with the challenges attached to procedural knowledge for further contexts.
In the future, it would be conceivable to use a shorter, and thus time-efficient, instrument by, for example, recording only one of the three fields of action. However, it is too early in the process of instrument development to recommend such a condensed version. Regarding the experts, there are eight significant differences out of 60 answers to the three fields of action. This might be explained by the interdisciplinary knowledge needed. However, it is still unclear how student teachers assess the different fields of action. Therefore, in order not to lose important information, the differentiation between the three fields of action will be kept. In our research, the measure of procedural knowledge is one of three parts to evaluate interdisciplinary knowledge of student teachers concerning biodiversity and climate change issues (see Section 1.2). The complete measurement instrument to evaluate situational, conceptual, and procedural knowledge will be applied in a survey with student teachers (starting in October 2018, n = 300; cf. Figure 1).
Based on the present study, recommendations on how to develop a measure for procedural knowledge, that reflects expert knowledge as well as student teacher perceptions, can be derived from the procedure demonstrated. The sophisticated procedure integrates a Delphi study and a think-aloud study. Furthermore, the relevant expert data for the benchmark (means, standard deviations, and ranking of effectiveness estimations) for using the presented measure for evaluating the procedural knowledge of student teachers are provided. Thus, the present study fills in the gap on how to assess SD-relevant procedural knowledge. In addition, to provide a refined procedure for developing a measure for procedural knowledge, the article specifically supplies a measure for procedural knowledge regarding the SD challenges biodiversity loss and climate change.

Author Contributions

L.R.-B. and S.B. conceived and designed the study. L.R.-B., I.G. and S.B. performed the study. L.R.-B., I.G. and S.B. analyzed the data. C.K. and N.A. contributed material and analyses (see Acknowledgments). L.R.-B., I.G. and S.B. wrote the paper. IG co-adviced parts of the study. S.B. coordinated and adviced the study.

Funding

This project is part of the “Qualitätsoffensive Lehrerbildung”, a joint initiative of the Federal Government and the Länder which aims to improve the quality of teacher training. The programme is funded by the Federal Ministry of Education and Research (reference number 01JA1617). The authors are responsible for the content of this publication.

Acknowledgments

The authors thank Christian Knedeisen for drafting solution strategies for the first Delphi round and for performing and analyzing parts of the first round in his Master of Education thesis. We thank Neele Achterberg for performing and analyzing the think-aloud study with student teachers in her Master of Education thesis.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

Scenario: Insects and pollination context
In Germany, 80% of the domestic crops and wild plants depend on insect pollination, e.g., honeybees and wild bees. Pollination is an example of an ecosystem service. Honeybees and wild bees use resources like nectar, pollen and water. Honeybees are bred by beekeepers and live in hives. Among others, wild bees nest in hedges, soils from extensive grassland, field margins or fallow land.
Until 2007, farms needed to fallow 10% of their land to receive subsidies from the EU’s common agricultural politics. The aim was reducing overproduction and soil erosion, as well as protecting biodiversity. In 2007, the set-aside instrument was abolished. Now, these areas are often used to cultivate plants for renewable energies.
Corn can be planted as a monoculture. By growing on the same fields over several years, corn becomes susceptible to pests and diseases. In 2007, farmers suffered from crop failures in Germany due to the western corn rootworm. Consequently, the Board of Agriculture suggested applying new seeds of a European seed company. These corn seeds were treated with an insecticide from the group of neonicotinoids. This so-called seed dressing should protect the seeds and later the entire plants from the corn rootworm.
In spring 2008, many bee colonies were lost in southern Germany, e.g., in the Upper Rhine valley, 11,500 bee colonies of 700 beekeepers were affected. The Julius-Kühn Institute in Braunschweig assumed that employing dressed corn seeds intoxicated honeybees and wild bees, and presumably other insects, too. Reasons for the bee intoxication were that the dressing did not stick to the corn seeds properly. Corn farmers frequently used dressed seeds to prevent suffering from crop loss. Corn is commonly grown for operating biogas facilities since it is a renewable resource. For honeybees and wild bees, those renewable resources are problematic as corn, for example, provides little food for bees. The European seed company, however, rather suspected the Varroa mite, that was introduced from Asia, as a cause for bee mortality. By breeding bees, comprising the aims of being easy to keep and producing a lot of honey, they became more vulnerable to the mite and other parasites and diseases.
In February 2018, the European Food Safety Authority (EFSA) confirmed the harmfulness of neonicotinoids. On 27 April 2018, the European Commission banned the use of the three neonicotinoids clothianidin, thiamethoxam and imidacloprid on fields, but not in greenhouses. Also, there are other neonicotinoids that may be used without restriction. The risk of these alternative neonicotinoids for bees is currently considered low. These plant protection products offer alternatives for farmers. However, environmental groups are calling for a ban on all neonicotinoids.

Appendix B

Scenario: Peatland use context
Worldwide, peatlands store 20–30% of total soil carbon although they only cover approximately 3% of the land surface. Peatlands characteristically feature a relatively high water level that can reach up to the surface. Because of the water saturation, organic matter is produced faster than it can be degraded. Peat is formed out of dead and preserved plant material over a long time span. The climate balance of undisturbed peatlands in Central Europe is roughly balanced.
The main types of peatlands are bogs and fens. Fens are fed by groundwater and show varying nutrient content. Bogs are only fed by rainwater and are rather poor nutrient systems. Peatlands provide ecosystem services. For example, they regulate water supplies, or they function as a recreation area. Due to little substance conversion, pollutants and nutrients are stored in peat. Using peatlands can have an impact on the ecosystem services.
In the past in Germany, peat from bogs was used as fuel. Nowadays, the German peat industry is the world’s largest producer of peat for potting soil. Many drained areas are now used for agriculture (grassland and cropland). In Germany, almost every peatland area is used. Only about 4% of German peatlands are in a near-natural, intact state. To use peatlands, the water level needs to be lowered. This is accomplished by drainage ditches and pumping stations. The decreased water level results in ventilating the peat. Decomposition processes are accelerated by the ventilation. As a result, the peat layers which have been formed for thousands of years, release carbon dioxide (CO2). Furthermore, nitrous oxide (N2O) can be emitted.
Though peatlands only constitute 6% of agricultural land, their use is responsible for 57% of all agricultural emissions. That equals 4.3% of Germany’s greenhouse gas emissions. Germany is committed to reducing greenhouse gas emissions. The aim until 2050 is to emit 80 to 95% less greenhouse gases than in the year 1990. At the UN conference on climate change in 2009 in Copenhagen, a limit of 2 °C of global warming was determined, compared to the temperature value of the years 1861–1880.
Renaturation measures of peatlands, such as raising the water level, can reduce greenhouse gas emissions; sometimes, even new peat can be formed. At the beginning of the renaturation measure, forced emissions of methane (CH4) can lead to higher greenhouse gas emissions than in a dehydrated state. However, over a long period peatlands can become carbon stores through renaturation.

Appendix C

Table
Table A1. Bonferroni-adjusted post hoc analysis.
Table A1. Bonferroni-adjusted post hoc analysis.
Solution StrategySustainable Land Use vs. Ecosystem ServicesSustainable Land Use vs. Biodiversity Conservation/Climate ProtectionEcosystem Services vs. Biodiversity Conservation/Climate Protection
MDp95% CIMDp95% CIMDp95% CI
IP-2−0.8670.001−10.36−0.376−0.05310.00−0.6740.5690.8160.0010.35110.28
IP-30.1580.248−0.0690.385−0.1050.992−0.3830.173−0.2630.062−0.5370.011
IP-80.3680.0450.0070.7300.3160.0300.0270.605−0.05310.00−0.3700.265
IP-9−0.3680.092−0.78300.46−0.4470.045−0.886−0.009−0.0790.992−0.2870.129
IP-100.2110.488−0.1710.592−0.2630.288−0.6590.132−0.4740.002−0.784−0.163
PU-60.2000.488−0.1610.5610.6000.0060.15710.0430.4000.085−0.0430.843
PU-80.3000.166−0.0860.6860.5000.0050.1440.8560.2000.775−0.2510.651
MD = mean difference; CI = confidence interval; IP = solution strategiesfor insects and pollination context; PU = solution strategies for peatland use context; tendencies.

References

  1. Bertschy, F.; Künzli, C.; Lehmann, M. Teachers’ Competencies for the Implementation of Educational Offers in the Field of Education for Sustainable Development. Sustainability 2013, 5, 5067–5080. [Google Scholar] [CrossRef] [Green Version]
  2. UNESCO (United Nations Educational, Scientific and Cultural Organization). Roadmap for Implementing the Global Action Programme on Education for Sustainable Development; UNESCO: Paris, France, 2014. [Google Scholar]
  3. Barth, M.; Rieckmann, M. State of the Art in Research on Higher Education for Sustainable Development. In Routledge Handbook of Higher Education for Sustainable Development; Barth, M., Michelsen, G., Rieckmann, M., Thomas, I., Eds.; Routledge: London, UK, 2016; pp. 100–113. ISBN 9780415727303. [Google Scholar]
  4. United Nations (UN), General Assembly. Transforming our World: The 2030 Agenda for Sustainable Development; United Nations (UN), General Assembly: New York, NY, USA, 2015. [Google Scholar]
  5. Lindemann-Matthies, P.; Constantinou, C.; Junge, X.; Köhler, K.; Mayer, J.; Nagel, U.; Raper, G.; Kadji-Beltran, C. The Integration of Biodiversity Education in the Initial Education of Primary School Teachers: Four Comparative Case Studies from Europe. Environ. Educ. Res. 2009, 15, 17–37. [Google Scholar] [CrossRef]
  6. Rieckmann, M.; Holz, V. Verankerung von BNE in der Lehrerbildung in Deutschland. Zeitschrift für Internationale Bildungsforschung und Entwicklungspädagogik (ZEP) 2017, 40, 4–10. [Google Scholar]
  7. Bourn, D.; Frances, H.; Phil, B. A Review of Education for Sustainable Development and Global Citizenship Education in Teacher Education, 2017. Background Paper Prepared for the 2017/8 Global Education Monitoring Report. Available online: http://unesdoc.unesco.org/images/0025/002595/259566e.pdf (accessed on 26 June 2018).
  8. LeNa. Teacher Education for a Sustainable Development from Pilot Projects and Initiatives to New Structures. A Memorandum on Reorienting Teacher Education in Germany, Austria and Switzerland. Available online: https://www.leuphana.de/fileadmin/user_upload/portale/netzwerk-lena/Memorandum_LeNa_English_Stand_August_15.pdf (accessed on 14 June 2018).
  9. Baumert, J.; Kunter, M. Stichwort: Professionelle Kompetenz von Lehrkräften. Zeitschrift für Erziehungswissenschaft 2006, 9, 469–520. [Google Scholar] [CrossRef]
  10. Baumert, J.; Kunter, M. Das Kompetenzmodell von COACTIV. In Professionelle Kompetenz von Lehrkräften. Ergebnisse des Forschungsprogramms COACTIV; Kunter, M., Baumert, J., Blum, W., Klusmann, U., Krauss, S., Neubrand, M., Eds.; Waxmann: Münster, Germany, 2011; pp. 29–53. ISBN 978-3-8309-2433-3. [Google Scholar]
  11. Reinke, V.; Hemmer, I. Bildung für nachhaltige Entwicklung—Über welche Kompetenzen verfügen Lehrkräfte und Akteur/-innen aus den außerschulischen Einrichtungen. Zeitschrift ZLB.KU 2017, 1, 38–42. [Google Scholar]
  12. Weinert, F.E. Concept of Competence: A Conceptual Clarification. In Defining and Selecting Key Competencies; Rychen, D.S., Salganik, L.H., Eds.; Horgrefe & Huber: Seattle, WA, USA; Toronto, ON, Australia; Bern, Switzerland; Göttingen, Germany, 2001; pp. 45–65. ISBN 978-0889372481. [Google Scholar]
  13. Shulman, L.S. Knowledge and Teaching: Foundation of the New Reform. Harv. Educ. Rev. 1987, 57. [Google Scholar] [CrossRef]
  14. Hellberg-Rode, G.; Schrüfer, G.; Hemmer, M. Brauchen Lehrkräfte für die Umsetzung von Bildung für nachhaltige Entwicklung (BNE) spezifische professionelle Handlungskompetenzen? Theoretische Grundlagen, Forschungsdesign und erste Ergebnisse. Zeitschrift für Geographiedidaktik (ZGD) 2014, 42, 257–281. [Google Scholar]
  15. Hellberg-Rode, G.; Schrüfer, G. Welche spezifischen professionellen Handlungskompetenzen benötigen Lehrkräfte für die Umsetzung von Bildung für Nachhaltige Entwicklung (BNE)? Ergebnisse einer explorativen Studie. Zeitschrift für Didaktik der Biologie 2016, 20, 1–29. [Google Scholar] [CrossRef]
  16. Fiebelkorn, F.; Menzel, S. Student Teachers’ Understanding of the Terminology, Distribution, and Loss of Biodiversity: Perspectives from a Biodiversity Hotspot and an Industrialized Country. Res. Sci. Educ. 2013, 43, 1593–1615. [Google Scholar] [CrossRef]
  17. Kaiser, F.; Fuhrer, U. Wissen für ökologisches Handeln. In Die Kluft zwischen Wissen und Handeln. Empirische und theoretische Lösungansätze; Mandl, H., Gerstenmaier, J., Eds.; Hogrefe: Göttingen, Germany, 2000; pp. 51–71. ISBN 9783801713386. [Google Scholar]
  18. Gräsel, C. Ökologische Kompetenz: Analyse und Förderung; Ludwig-Maximilian-Universität, Fakultät für Psychologie und Pädagogik: München, Germany, 2000; in press. [Google Scholar]
  19. Anderson, L.W.; Krathwohl, D.R.; Airasian, P.W.; Cruikshank, K.A.; Mayer, R.E.; Pintrich, P.R.; Raths, J.; Wittrock, M.C. A Taxanomy for Learning, Teaching, and Assessing. A Revision of Bloom’s Taxanomy of Educational Objectives; David McKay Company, Inc.: New York, NY, USA, 2001; ISBN 0-8013-1903-X. [Google Scholar]
  20. Alexander, P.A.; Schallert, D.L.; Hare, V.C. Coming to Terms: How Researchers in Learning and Literacy talk about Knowledge. Rev. Educ. Res. 1991, 61, 315–343. [Google Scholar] [CrossRef]
  21. Ryle, G. The Concept of the Mind; University of Chicago Press: Chicago, IL, USA, 1949. [Google Scholar]
  22. Anderson, J.R. Acquisition of Cognitive Skill. Psychol. Rev. 1982, 89, 369–406. [Google Scholar] [CrossRef]
  23. De Jong, T.; Ferguson-Hessler, M. Types and Qualities of Knowledge. J. Educ. Psychol. 1996, 31, 105–113. [Google Scholar] [CrossRef]
  24. Sáenz, C. The Role of Contextual and Procedural Knowledge in Activating Mathematical Competencies (PISA). Educ. Stud. Math. 2009, 71, 123–143. [Google Scholar] [CrossRef]
  25. Baroody, A.J.; Feil, Y.; Johnson, A.R. An Alternative Reconceptualization of Procedural and Conceptual Knowledge. J. Res. Math. Educ. 2007, 38, 115–131. [Google Scholar]
  26. Rittle-Johnson, B.; Koedinger, K.R. Designing Knowledge Scaffolds to Support Mathematical Problem Solving. Cognit. Instruct. 2005, 23, 313–349. [Google Scholar] [CrossRef]
  27. Koch, S.; Barkmann, J.; Strack, M.; Sundawati, L.; Bögeholz, S. Knowledge of Indonesian University Students on the Sustainable Management of Natural Resources. Sustainability 2013, 5, 1443–1460. [Google Scholar] [CrossRef] [Green Version]
  28. Robertson, M. Sustainability Principles and Practice; Taylor and Francis: Hoboken, NJ, USA, 2014; ISBN 041584018X. [Google Scholar]
  29. United Nations (UN). The Sustainable Development Goals Report 2017. Available online: https://unstats.un.org/sdgs/files/report/2017/TheSustainableDevelopmentGoalsReport2017.pdf (accessed on 28 June 2018).
  30. Esa, N. Environmental Knowledge, Attitude and Practices of Student Teachers. Int. Res. Geogr. Environ. Educ. 2010, 19, 39–50. [Google Scholar] [CrossRef]
  31. Gayford, C. Biodiversity Education: A Teacher’s Perspective. Environ. Educ. Res. 2000, 6, 347–361. [Google Scholar] [CrossRef]
  32. Summers, M.; Kruger, C.; Childs, A.; Mant, J. Primary School Teachers’ Understanding of Environmental Issues: An Interview Study. Environ. Educ. Res. 2000, 6, 293–312. [Google Scholar] [CrossRef]
  33. Dikmenli, M. Biology Student Teachers’ Conceptual Frameworks in Regarding Biodiversity. Education 2010, 130, 479–490. [Google Scholar]
  34. Khalid, T. Pre-service Teachers’ Misconceptions Regarding Three Environmental Issues. Can. J. Environ. Educ. 2001, 6, 102–120. [Google Scholar]
  35. Boon, H.J. Climate Change? Who Knows? A Comparison of Secondary Students and Pre-service Teachers. Aust. J. Teach. Educ. 2010, 35, 103–120. [Google Scholar] [CrossRef]
  36. Boon, H.J. Teachers and the Communication of Climate Change Science: A Critical Partnership in Australia. Procedia—Soc. Behav. Sci. 2014, 116, 1006–1010. [Google Scholar] [CrossRef]
  37. Bögeholz, S.; Hössle, C.; Höttecke, D.; Menthe, J. Bewertungskompetenz. In Theorien in der Naturwissenschaftsdidaktischen Forschung; Krüger, D., Parchmann, I., Schecker, H., Eds.; Springer: Berlin/Heidelberg, Germany, 2018; pp. 261–281. ISBN 978-3-662-56319-9. [Google Scholar]
  38. Schreiber, J.-R. Competencies, Themes, Standards, Design of Lessons and Curricula. In Curriculum Framework: Education for Sustainable Development, 2nd ed.; Schreiber, J.-R., Siege, H., Eds.; Engagement Global gGmbH: Bonn, Germany, 2016; pp. 86–110. ISBN 978-3-06-230062-2. [Google Scholar]
  39. Overwien, B. Education for Global and Sustainable Development in Teacher Education. In Curriculum Framework: Education for Sustainable Development, 2nd ed.; Schreiber, J.-R., Siege, H., Eds.; Engagement Global gGmbH: Bonn, Germany, 2016; pp. 420–435. ISBN 978-3-06-230062-2. [Google Scholar]
  40. Gayford, C. Controversial Environmental Issues. A Case Study for the Professional Development of Science Teachers. Int. J. Sci. Educ. 2002, 24, 1191–1200. [Google Scholar] [CrossRef]
  41. Potts, S.G.; Biesmeijer, J.C.; Kremen, C.; Neumann, P.; Schweiger, O.; Kunin, W.E. Global Pollinator Declines: Trends, Impacts and Drivers. Trends Ecol. Evol. 2010, 25, 345–353. [Google Scholar] [CrossRef] [PubMed]
  42. Klein, A.-M.; Vaissière, B.E.; Cane, J.H.; Steffan-Dewenter, I.; Cunningham, S.A.; Kremen, C.; Tscharntke, T. Importance of pollinators in changing landscapes for world crops. Proc. Biol. Sci. 2007, 274, 303–313. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  43. Gallai, N.; Salles, J.-M.; Settele, J.; Vaissière, B.E. Economic valuation of the vulnerability of world agriculture confronted with pollinator decline. Ecol. Econ. 2009, 68, 810–821. [Google Scholar] [CrossRef] [Green Version]
  44. European Academies’ Science Advisory Council (EASAC). Ecosystem Services, Agriculture and Neonicotinoids; EASAC Policy Report 26; EASAC Secretariat Deutsche Akademie der Naturforscher Leopoldina: Halle, Germany, 2015. [Google Scholar]
  45. Hopfenmuller, S.; Steffan-Dewenter, I.; Holzschuh, A. Trait-specific Responses of Wild Bee Communities to Landscape Composition, Configuration and Local Factors. PLoS ONE 2014, 9, 1–10. [Google Scholar] [CrossRef] [PubMed]
  46. Krewenka, K.M.; Holzschuh, A.; Tscharntke, T.; Dormann, C.F. Landscape Elements as Potential Barriers and Corridors for Bees, Wasps and Parasitoids. Biol. Conserv. 2011, 144, 1816–1825. [Google Scholar] [CrossRef]
  47. Henry, M.; Beguin, M.; Requier, F.; Rollin, O.; Odoux, J.-F.; Aupinel, P.; Aptel, J.; Tchamitchian, S.; Decourtye, A. A Common Pesticide Decreases Foraging Success and Survival in Honey Bees. Science 2012, 336, 348–350. [Google Scholar] [CrossRef] [PubMed]
  48. Bundesministerium für Umwelt, Naturschutz und Nukleare Sicherheit (BMUB). Die Biene: Eines der wichtigsten Nutztiere. Umwelt im Unterricht: Materialien und Service für Lehrkräfte—BMUB-Bildungsservice. 2013. Available online: http://ag-binnenduene.de/downloads/130502_Bienen.pdf (accessed on 5 July 2016).
  49. Pistorius, J.; Bischoff, G.; Heimbach, U.; Stähler, M. Bee poisoning incidents in Germany in spring 2008 caused by abrasion of active substance from treated seeds during sowing of maize. In Julius-Kühn-Archiv: Vol. 423. Hazards of Pesticides to Bees. 10th International Symposium of the ICP-BR Bee Protection Group Bucharest (Romania); Oomen, P.A., Thompson, H.M., Eds.; Julius Kühn-Institut: Quedlinburg, Germany, 2008; pp. 118–126. [Google Scholar]
  50. Ernährung und Landwirtschaft (BMELV). Bestandsaufnahme und Perspektiven der Bienenhaltung und Imkerei in Deutschland; Bundesministerium für. 2013. Available online: https://www.bmel.de/SharedDocs/Downloads/Tier/TierzuchtTierhaltung/Bestandsaufnahme-Imkerei.pdf?__blob=publicationFile (accessed on 19 July 2016).
  51. Christensen, T.R.; Friborg, T.; Byrne, K.A.; Chojnicki, B.; Christensen, T.R.; Freibauer, A.; Friborg, T.; Frolking, S.; Lindroth, A.; Mailhammer, J.; et al. EU Peatlands: Current Carbon Stocks and Trace Gas Fluxes. In Report 4/2004 to ‘Concerted Action: Synthesis of the European Greenhouse Gas Budget’; Geosphere-Biosphere Centre, University of Lund: Lund, Sweden, 2004. [Google Scholar]
  52. Frolking, S.; Talbot, J.; Jones, M.C.; Treat, C.C.; Kauffman, J.B.; Tuittila, E.-S.; Roulet, N. Peatlands in the Earth’s 21st Century Climate System. Environ. Rev. 2011, 19, 371–396. [Google Scholar] [CrossRef]
  53. Romine, W.L.; Sadler, T.D.; Kinslow, A.T. Assessment of Scientific Literacy: Development and Validation of the Quantitative Assessment of Socio-Scientific Reasoning (QuASSR). J. Res. Sci. Teach. 2017, 54, 274–295. [Google Scholar] [CrossRef]
  54. Mang, J.; Ustjanzew, N.; Schiepe-Tiska, A.; Prenzel, M.; Sälzer, C.; Müller, K.; Gonzaléz Rodríguez, E. PISA 2012 Skalenhandbuch: Dokumentation der Erhebungsinstrumente; Waxmann: Münster, Germany; New York, NY, USA, 2018; ISBN 978-3-8309-3825-5. [Google Scholar]
  55. Derksen, C. Situationales und konzeptuelles Wissen zu “Biodiversität” von Lehramtsstudierenden: Eine Studie Lauten Denkens zur Weiterentwicklung eines quantitativen Messinstruments. Master’s Thesis, Georg-August-Universität Göttingen, Göttingen, Germany, 14 September 2017. [Google Scholar]
  56. Grube, Y. Eine Studie Lauten Denkens am Beispiel von “Klimawandel und Moornutzung” zur Weiterentwicklung eines Messinstruments. Master’s Thesis, Georg-August-University, Göttingen, Germany, 15 March 2018. [Google Scholar]
  57. Richter-Beuschel, L.; Derksen, C.; Bögeholz, S. Konzeptuelles Wissen angehender Lehrkräfte für Bildung für Nachhaltige Entwicklung. In Treffpunkt Biologische Vielfalt XVI—Interdisziplinärer Forschungsaustausch im Rahmen des Übereinkommens über die biologische Vielfalt—BfN-Skripten 487; Korn, H., Dünnfelder, H., Schliep, R., Eds.; Bundesamt für Naturschutz (BfN): Bonn, Germany, 2018; pp. 88–95. [Google Scholar]
  58. Knedeisen, C. Maßnahmen Gegen Bienensterben und Moordegradierung. Eine Delphi-Studie zum Prozeduralen Wissen. Master‘s Thesis, Georg-August-University, Göttingen, Germany, 6 September 2017. [Google Scholar]
  59. Häder, M. Delphi-Befragungen. Ein Arbeitsbuch; Westdeutscher Verlag: Wiesbaden, Germany, 2002; ISBN 978-3-658-01928-0. [Google Scholar]
  60. Niederberger, M.; Renn, O. Das Gruppendelphi-Verfahren. Vom Konzept bis zur Anwendung; Springer: Wiesbaden, Germany, 2018; ISBN 978-3-658-18755-2. [Google Scholar]
  61. Mayring, P. Qualitative Inhaltsanalyse: Grundlagen und Techniken; Beltz: Weinheim, Germany, 2007; ISBN 9783825282295. [Google Scholar]
  62. Ericsson, K.A.; Simon, H.A. Protocol Analysis: Verbal Reports as Data. A Bradford Book; MIT Press: London, UK, 1993; ISBN 9780262050470. [Google Scholar]
  63. Sandmann, A. Lautes Denken—Die Analyse von Denk-, Lern- und Problemlöseprozessen. In Methoden in der Naturwissenschaftsdidaktischen Forschung; Krüger, D., Parchmann, I., Schecker, H., Eds.; Springer Spektrum: Berlin, Germany, 2014; pp. 179–188. ISBN 978-3-642-37826-3. [Google Scholar]
  64. Achterberg, N. Prozedurales Wissen von Lehramtsstudierenden zur Gestaltung Nachhaltiger Entwicklung. Erkenntnisse aus einer Studie Lauten Denkens für die Weiterentwicklung eines Messinstrumentes. Master’s Thesis, Georg-August-University, Göttingen, Germany, 16 March 2018. [Google Scholar]
  65. Vallor, R.; Yates, K.; Brody, M. Delphi Research Methodology Applied to Place-Based Watershed Education. Educ. Sci. 2016, 6. [Google Scholar] [CrossRef]
  66. Vorgrimler, D.; Wübben, D. Die Delphi-Methode und ihre Eignung als Prognoseinstrument. Wirtschaft und Statistik 2003, 8, 763–774. [Google Scholar]
  67. Cohen, J. Statistical Power Analysis for the Behavioral Sciences, 2nd ed.; Lawrence Erlbaum Associates: New York, NY, USA, 1988; ISBN 9780805802832, 0805802835. [Google Scholar]
  68. Bortz, J.; Döring, N. Forschungsmethoden und Evaluation: Für Human- und Sozialwissenschaftler, 3rd ed.; Springer: Berlin/Heidelberg, Germany, 2002; ISBN 978-3-662-07299-8. [Google Scholar]
  69. Lenske, G.; Thillmann, H.; Wirth, J.; Dicke, T.; Leutner, D. Pädagogisch-psychologisches Professionswissen von Lehrkräften: Evaluation des ProwiN-Tests. Zeitschrift für Erziehungswissenschaften 2015, 18, 225–245. [Google Scholar] [CrossRef]
  70. Gordon, T.; Pease, A. RT Delphi: An Efficient, “Round-less” Almost Real Time Delphi Method. Technol. Forecast. Soc. Chang. 2006, 73, 321–333. [Google Scholar] [CrossRef]
Education 08 00190 g001 550
Figure 1. Procedure for developing a measure for procedural knowledge exemplified for solution strategies regarding Sustainable Development (SD) challenges.
Figure 1. Procedure for developing a measure for procedural knowledge exemplified for solution strategies regarding Sustainable Development (SD) challenges.
Education 08 00190 g001
Education 08 00190 g002 550
Figure 2. Assessment of procedural knowledge (1 = insect and pollination context; 2 = peatland use context; IP = solution strategies for insect and pollination context; PU = solution strategies for peatland use context; = the same applies to the other solution strategies of these contexts).
Figure 2. Assessment of procedural knowledge (1 = insect and pollination context; 2 = peatland use context; IP = solution strategies for insect and pollination context; PU = solution strategies for peatland use context; = the same applies to the other solution strategies of these contexts).
Education 08 00190 g002
Education 08 00190 g003 550
Figure 3. Examples of graphically processed results of quantitative data (percentage distribution and median) of two items of the insects and pollination context of the first Delphi round (ineffective to very effective: Education 08 00190 i001 Education 08 00190 i002 Education 08 00190 i003 Education 08 00190 i004; median: Education 08 00190 i005).
Figure 3. Examples of graphically processed results of quantitative data (percentage distribution and median) of two items of the insects and pollination context of the first Delphi round (ineffective to very effective: Education 08 00190 i001 Education 08 00190 i002 Education 08 00190 i003 Education 08 00190 i004; median: Education 08 00190 i005).
Education 08 00190 g003
Education 08 00190 g004 550
Figure 4. Means of solution strategies regarding the insect and pollination context (unweighted effectivity estimations). * = significant differences between the fields of action.
Figure 4. Means of solution strategies regarding the insect and pollination context (unweighted effectivity estimations). * = significant differences between the fields of action.
Education 08 00190 g004
Education 08 00190 g005 550
Figure 5. Means of solution strategies regarding the peatland use context (unweighted effectivity estimations). * = significant differences between the fields of action.
Figure 5. Means of solution strategies regarding the peatland use context (unweighted effectivity estimations). * = significant differences between the fields of action.
Education 08 00190 g005
Table
Table 1. Sample composition of the Delphi survey.
Table 1. Sample composition of the Delphi survey.
Working Areas (Age Groups)
UniversityESD—Geography Education (41–50), Geography Education (41–50), Geography Education (41–50), Biology Education (41–50), Biology and Geography Education (51–60), Science Education (31–40), Political Science Education (41–50)
Biology (51–60) 2, Plant Ecology (>60) 2, Paleoecology and Botany (51–60)
Peatlands and Paleoecology (>60) 2, Climatology (41–50)
Soil Science (51–60), Agroecology (21–30) 1, −2, Agroecology (31–40), Agroecology (>60) 2, Agricultural Economics (41–50)
Human Geography—Sustainable Resource Use (51–60)
Risk and Sustainability Research (41–50)
Non-university InstitutionsResearch Institute: Greenhouse gas emissions of organic soils, policy advice in the field of climate-friendly use of organic soil (31–40)
Professional Association: Representative of beekeepers (>60) 1
Ministry of Environment: Conservation management (peatland protection) (41–50), protection of species (31–40) 1
ESD = Education for Sustainable Development; 1 these experts participated in the first Delphi round only; 2 these experts participated in second Delphi round only; −2 excluded in the second Delphi round.
Table
Table 2. Weighted effectiveness estimations of solution strategies regarding the insects and pollination context (M: mean; SD: standard deviation) for total weighted effectivity with respect to SD challenge (“SD effectivity”) and separated into three fields of action (n = 19).
Table 2. Weighted effectiveness estimations of solution strategies regarding the insects and pollination context (M: mean; SD: standard deviation) for total weighted effectivity with respect to SD challenge (“SD effectivity”) and separated into three fields of action (n = 19).
Solution Strategies Regarding Insects and PollinationSD Effectivity (Mean Across Fields of Action) Fields of ActionOverarching Topic
Sustainable Land UseEcosystem ServicesBiodiversity Conservation
MSDMSDMSDMSD
IP-1 Align the breeding of honeybees for resistance to disease and parasites.2.210.551.870.852.850.701.910.76Research for sustainable development
IP-2 Individuals ask in petitions to introduce bee-friendly laws.2.230.712.140.822.240.782.310.85Environmental policy
IP-3 Include contents of pollinator respective bee-related problems in curricula for schools, environment-related vocational training, and university studies.2.541.002.251.032.411.092.690.97Education for sustainable development
IP-4 The government provides financial incentives for using bee-friendly bloomers as biomass in biogas power stations.2.710.742.680.962.680.692.770.66Agricultural policy
IP-5 Support pollinator-friendly agriculture by purchasing ecologically produced products. 2.930.712.880.683.000.782.930.85Sustainable consumption
IP-6 The legislator prohibits application of neonicotinoids.3.110.823.031.043.030.813.270.81Environmental/agricultural policy
IP-7 Farmers reduce their use of pesticides and fertilizers. 3.480.563.560.603.310.663.580.68Sustainable production
IP-8 Realign agricultural subsidies to stop promoting conventional and intensive agriculture.3.550.423.770.443.420.603.470.52Agricultural policy, sustainable production
IP-9 Design a cultural landscape to serve pollinators as food source and habitat.3.620.373.600.513.420.603.850.37Sustainable management, diversity of species
IP-10 Strengthen the protection of wild bees and other pollinating insects.3.710.513.490.883.780.433.860.36Diversity of species, sustainable manage ment/production, environmental policy
Lowest effectiveness, highest effectiveness, IP-6 to be excluded for the benchmark for evaluating student teacher knowledge (for explanation see Section 4).
Table
Table 3. Weighted effectiveness estimations of solution strategies regarding peatland use context (M: mean, SD: standard deviation) for total weighted effectivity with respect to SD challenge (“SD effectivity”) and separated into the three fields of action (n = 20).
Table 3. Weighted effectiveness estimations of solution strategies regarding peatland use context (M: mean, SD: standard deviation) for total weighted effectivity with respect to SD challenge (“SD effectivity”) and separated into the three fields of action (n = 20).
Solution Strategies Regarding Peatland UseSD Effectivity (Mean Across Fields of Action)Fields of ActionOverarching Topic
Sustainable Land UseEcosystem ServicesClimate Protection
MSDMSDMSDMSD
PU-1 After rewetting of intensively agricultural used peatlands, farmers grow moisture-loving plants, e.g., reed.2.540.812.750.962.190.912.690.92Sustainable management/production
PU-2 Individuals purchase products only from sustainable peat extraction.2.690.872.780.952.610.882.690.96Sustainable consumption
PU-3 Inform the public more intensively about the important role of peatlands, e.g., via media or educational projects.2.700.752.750.712.570.852.780.86Education for sustainable development
PU-4 Allow companies to incorporate CO2 savings from peatland conservation into the EU emissions trading.2.970.693.110.802.880.822.920.68Environmental policy
PU-5 Investigate cultivation methods that preserve peatlands to apply them on agricultural-used peatlands.2.980.553.090.672.910.642.940.60Research for SD, sustainable management
PU-6 Cultivate peatlands without fertilizers and pesticides.2.980.923.200.923.080.872.671.20Sustainable production
PU-7 Raise the water level of dehydrated peatlands to the water level of intact, near-nature peatlands.3.230.623.080.823.130.873.480.77Sustainable management
PU-8 Intensify the investigation of regenerative peat substitutes.3.260.713.340.693.130.863.310.76Research for sustainable development
PU-9 Apply existing lawsstricter, e.g., prohibit the converting of grassland into maize cultivation.3.270.693.500.713.280.653.030.97Agricultural policy
PU-10 Provide agricultural subsidies only for sustainably managed peatlands.3.500.643.640.643.390.723.470.71Environmental policy
Lowest effectiveness, highest effectiveness, PU-2 to be excluded for the benchmark for evaluating student teacher knowledge (for explanation see Section 4).
Table
Table 4. Differences in the effectivity estimations of solution strategies for insects and pollination and peatland use contexts using repeated measures (rm) ANOVA.
Table 4. Differences in the effectivity estimations of solution strategies for insects and pollination and peatland use contexts using repeated measures (rm) ANOVA.
Insects and PollinationPeatland Use
ItemrmANOVApPartial η²ItemrmANOVApPartial η²
IP-1F (2, 36) = 11.730.0010.395PU-1 1F (1.47, 26.36) = 3.230.0690.152
IP-2 1F (1.33, 24.00) = 0.2620.6810.014PU-2 1F (1.47, 26.52) = 0.8690.4000.046
IP-3F (2, 36) = 3.600.0380.167PU-3F (2, 38) = 2.080.1390.099
IP-4 1F (1.47, 26.48) = 0.6550.4830.035PU-4F (2, 38) = 0.9030.4140.045
IP-5F (2, 36) = 0.4860.6190.026PU-5 1F (1.55, 29.44) = 0.7190.4620.036
IP-6F (2, 36) = 2.220.1240.110PU-6F (2, 38) = 7.390.0020.280
IP-7F (2, 36) = 2.730.0790.132PU-7 1F (1.45, 27.55) = 2.060.1560.098
IP-8F (2, 36) = 5.270.0100.226PU-8F (2, 36) = 2.520.0950.123
IP-9F (2, 36) = 5.900.0060.247PU-9F (2, 38) = 5.470.0080.224
IP-10F (2, 36) = 5.850.0060.245PU-10 1F (1.38, 26.15) = 2.870.0910.131
F (df, dferror) = F-value, 1 Greenhouse–Geisser correction.
Table
Table 5. Reliability of the scales with unweighted effectivity estimations of solutions strategies (n = 20).
Table 5. Reliability of the scales with unweighted effectivity estimations of solutions strategies (n = 20).
Sustainable Land UseEcosystem ServicesBiodiversity ConservationClimate Protection
Insects and pollination0.8380.800 *0.8870.874 *0.7580.734 *
Peatland use0.5720.456 *0.7630.727 * 0.7640.726 *
(0.633)(0.513) * (0.714) * (0.780)
Both contexts0.8520.810 *0.9060.889 *
* = without item IP-6 or PU-2; brackets = reliability analyses weighted by averaged certainty, only reported if deviation > ± 0.01.

Share and Cite

MDPI and ACS Style

Richter-Beuschel, L.; Grass, I.; Bögeholz, S. How to Measure Procedural Knowledge for Solving Biodiversity and Climate Change Challenges. Educ. Sci. 2018, 8, 190. https://doi.org/10.3390/educsci8040190

AMA Style

Richter-Beuschel L, Grass I, Bögeholz S. How to Measure Procedural Knowledge for Solving Biodiversity and Climate Change Challenges. Education Sciences. 2018; 8(4):190. https://doi.org/10.3390/educsci8040190

Chicago/Turabian Style

Richter-Beuschel, Lisa, Ingo Grass, and Susanne Bögeholz. 2018. "How to Measure Procedural Knowledge for Solving Biodiversity and Climate Change Challenges" Education Sciences 8, no. 4: 190. https://doi.org/10.3390/educsci8040190

APA Style

Richter-Beuschel, L., Grass, I., & Bögeholz, S. (2018). How to Measure Procedural Knowledge for Solving Biodiversity and Climate Change Challenges. Education Sciences, 8(4), 190. https://doi.org/10.3390/educsci8040190

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop