Assessing Teachers’ Capabilities to Work with Models and Evaluate Results in the Context of a Complex and Authentic STEM Problem

Abstract

Since the practice turn, the contemporary education landscape has been shifting from mere knowledge dissemination to empowering students to solve problems. Special emphasis is given to problems on which students work for an extended period (at least several hours; frequently multiple school days). While working on such problems, it is essential to employ a variety of activities. Two of these are working with models and evaluating models and their results. One topic that has received little attention up to now is the question of to what extent educators are able to apply these skills. This study, fundamentally exploratory in nature, seeks to delve into such an assessment by evaluating the competence of $n=20$ educators in estimating and evaluating building evacuation duration using digital simulations. Our results show that the participants self-assessed as being able to solve such exercises. However, this was contrasted by our external assessment of the solutions provided by the participants, which showed that the solutions lacked in quality.

1. Introduction

The practice turn [1] describes a shift in modern educational paradigms, especially in regard to two core aspects. The first is the shift from teacher-centered to student-centered instructions [2,3], and the second is a change in focus, from teaching factual knowledge “to helping students figure out phenomena and design solutions to problems” [4]. To implement these new paradigms in practice, a multitude of approaches have been proposed in recent years. These include, among others, scenario-based education [5,6,7], situated learning [8,9], complex and authentic modelling [10,11], problem-based learning [12,13], and simulation-based-learning [14,15].

A unifying factor between all of these approaches is the fact that students typically work on problems for an extended period of time—at least hours, frequently multiple days. Thus, the corresponding exercises form a “sharp contrast to many other modelling activities common at school” [11] and that are (currently) often only included in regular school lessons that follow a 45-minute format. Instead, where already implemented, other formats are used. These include so-called modeling weeks in Germany, project days in (some) schools focusing on solving a single problem (c.f. [11,16]), or teaching–learning-labs at universities (c.f. [17,18]). We call these complex and authentic problems, based on two key characteristics that are frequently demanded for problems in such an educational context.

Based on the assumption that the implementation of such approaches into regular school education is indeed desirable, one has to ask whether current educators are equipped to teach in such settings. As such, the central goal of this paper was an exploratory analysis of educators’ abilities to work with problems that could be used in such a setting—a necessary prerequisite to be able to teach with them (c.f. [19]). More precisely, we focus our qualitative evaluation on two activities that are crucial for any problem-solving process: working with models, and evaluating models and the results achieved using them. Additionally, we analyzed whether the self-assessment of the educators regarding being able to solve such exercises was aligned with our external assessment.

We approached this measurement with an evaluative qualitative analysis of n = 20 educators tasked to solve a problem we posed. Our results indicated that, while our participants indeed generally felt able to solve such problems and teach with them, they nevertheless achieved low evaluation scores for both activities.

2. Educational Background

There are many different problems that have been proposed for the approaches listed previously. For example, twenty published examples recommended for complex and authentic modelling are presented in (p. 291, [11]).

In this section, we first describe STEM problems and teaching about problem-solving in general. Then, we explore two characteristics that are frequently demanded of problems in the described context: complexity and authenticity. Lastly, we outline the two activities chosen for this evaluation that are crucial for approaching any such problem: working with models, and evaluating models and the results achieved using them. Lastly, we introduce the SOLO taxonomy we use later to assess the quality of the argumentation used in the evaluation.

2.1. Problem-Solving in STEM Education

Problems in education are defined as “tasks that cannot be solved by direct effort” [20], emphasizing the necessity for strategic and structured approaches to finding solutions [21].

For STEM problems, a common distinction is made in regard to the relevance of each of the individual STEM subjects to the problem stated [22]. In an integrated approach, “[a]ll four disciplines are combined equally in a real-world problem” [23]. Contrary to that, in a separate approach, “each discipline is taught separately as an independent subject, with little or no integration” [23]. The problem chosen for this work uses a semi-integrated approaches. More precisely, our problem emphasizes only two of the STEM subjects (mathematics and technology), rather than all of them. Notably, while this approach is common, it is not without criticisms (c.f. [22,23,24]).

2.1.1. Teaching Problem-Solving

Teachers are required to possess a multifaceted range of knowledge and skills, some of which were identified in a literature review: to teach problem-solving, one needs knowledge of problems, problem-solving, problem posing, students as problem solvers, problem-solving instructions, and affective factors and beliefs [25]. Out of those, in-depth understanding and firsthand experience in problem-solving are pivotal for effectively imparting these skills to students, as teachers should “experience mathematical problem-solving from the perspective of the problem solver before they can adequately deal with its teaching” [19].

Notably, prior research has already identified difficulties teachers encounter in problem-solving, including an “inability to successfully relate the solutions to real life”, a “lack of flexibility in choice of problem-solving approaches”, an “inflexibility in their choice or management of problem-solving strategies”, and “a lack of strategies for interpreting the information given to them in word problems and for recognising the appropriate procedure to use” (c.f. [25] for more comprehensive list).

However, most of the prior research focused either on smaller-scale problems or modelling tasks (e.g., [26,27,28]), or on relative gains from interventions (e.g., [29,30,31]), rather than an overall assessment of the level of competence shown by educators working on such larger-scale problems as described earlier.

Such a limited analysis of problem-solving activities is problematic, as “[problem-solving] proficiency is not a one-dimensional concept and cannot be achieved by focusing on just one or two of the factors that define it” [25]. As such, assessment of the many different sub-activities necessary for problem-solving is desirable for a comprehensive picture of the abilities of certain groups to solve problems. Such assessments of teachers’ capabilities are especially important, as “assessment of mathematics teachers’ knowledge [is] one of the most important parameters of the quality of mathematics teaching in school” [32].

Notably, as teachers must decide which limited aspects of the problem should be simplified in order to be suitable for the target audience, their problem-solving proficiency needs to be higher for teaching problem-solving than for solving problems themselves. Furthermore, “the teacher needs to have a broad and deep understanding of the diversity of approaches that students might take. Trying to quickly grasp the mathematics [...] while simultaneously devising appropriate responses, is not an easy task for the teacher. The difficulties in doing this should not be underestimated” [33].

2.1.2. Complex Problems

Complex problems are characterized by five key properties: the large numbers of variables describing the situation, interdependence among these variables, unknown variables and goals, changes in the situation under consideration, and the presence of multiple (possibly conflicting) goals [34]. Notably, the last two properties are only frequently associated with complex problems, but are not a necessary ingredient for them [34]. However, in education where problems are artificially introduced by the teacher, they do not naturally change over time or possess multiple antagonistic goals. As solving complex problems already entails sufficient difficulties (c.f. [35]), the problem we pose will not have these two properties. The importance of solving complex problems is underscored by their critical role in the 21st-century workplace [36], and solving them is frequently considered one of the most important 21st century skills (e.g., [37,38,39,40,41]).

2.1.3. Authenticity in STEM Education

Many authors have highlighted the significance of authenticity in STEM education (e.g., [10,42,43,44]), where the use of real-world contexts is crucial to contextualize learning and spur student engagement [45]. In fact, some even argue that “authenticity must be viewed as a cornerstone of STEM literacy problems” [46].

Authenticity, although a prevalent concept, varies in interpretation: “[T]he uses vary greatly from referring to externally defined practices to student relevance” [47]. Generally, authenticity refers to an “alignment of student learning experiences with the world for which they are being prepared for” [10]. However, more precisely, eight different ways of using the term have been identified [48].

One usage refers to authentic problems that should “be grounded in the world of the students” [47]. This emphasizes the importance of sense-making, i.e., the ability of students “to make sense of the phenomena they encounter and understand the relationships among them, to have a working knowledge of the world beyond the horizon of their own limited experience” [49]. As such, authentic problems “shall articulate the relevance of mathematics in daily life, environment and sciences and impart competencies to apply mathematics in daily life, environment and sciences” [11].

Another usage focuses on authentic activities: activities performed by students that “have real world relevance” [44]. Such activities can be “comparable with practices of professional scientists” [47], “comparable with non-professional citizen practices” [47], or “relevant and real to workplace situations” [50].

Empirically, authenticity has been shown to have the potential to enable desirable educational outcomes [51], including increases in motivation [52,53,54] and task performance [52,54,55], as well as improved collaboration [56].

2.2. Models and Working with Models in Problem-Solving

Models play a central role in STEM problem-solving (c.f. [21,57,58]). Their significance is notably underscored in steps five (building models), seven (using models, computing), and eight (generative data, potentially with a simulation) of the problem-solving process presented in [13].

2.2.1. Defining Models and Their Common Properties

In the context of our research, a model is defined as a simplified, conceptual representation of a system in the real world; their description frequently utilizes images, numbers, formulas, programs, and their interrelationships [58].

To enhance clarity, this article introduces an expansive use of the term model, distinguishing among abstract, parameterized, and implemented models—as illustrated in Figure 1: To enable a mathematical approach to a problem, a real-world problem (e.g., estimating the time to evacuate a building) can be modelled using an abstract model (e.g., a grid automaton). This abstract model can then be applied to a specific real-world situation (e.g., estimating the time to evacuate a given sports hall) by selecting a suitable parameter (e.g., two cells for the width of the hallway in this sports hall), leading to a parameterized model. In practice, the calculations of a model are frequently automated with technology. In this case, the abstract and parameterized model have to be implemented using technology, leading to an implemented model. This distinction is especially important when discrepancies arise between parameterized and implemented models due to implementation errors.

The transition from real-world scenarios to any model type is termed model-building [59], encompassing activities like simplifying and structuring a situation to construct a model. The finished model should “on the one hand still contain essential features of the original situation, but is on the other hand already so schematized that (if at all possible) it allows for an approach with mathematical means” [60]. Models can be used to “provide information (or predictions) on some characteristics of [the model] which are not explicitly stated during its elaboration” [61] (predictive models). The extent to which a model can generate novel insights is termed its utility [61], with its usability being a measure of the effort needed to glean these insights [62]. However, other types of model exist: “descriptive models are also ubiquitous in science education materials” [63] and explanatory models are central to science research [63].

2.2.2. Selection of Models

The necessity for models to be self-consistent [61] does not preclude the use of different models to represent the same situation. This variability can be used to fulfill different goals of a model [64] but also necessitates a conscious selection and subsequent evaluation of a chosen model and its parameters: “models may select different features of the object, because there is a different evaluation of what characterizes the object, or because there are distinct aspects of the object which deserve modeling” [61].

Notably, the predictions of different models can contradict each other, highlighting the relevance of model evaluation. Additionally, within classroom settings, teachers should not only be adept at solving problems with one model but should guide students in model selection, problem-solving, and result evaluation, independently of the students’ model choices [33].

2.2.3. Working with Technological Models

Technological interaction is often indispensable when working with models, as “both the design and interpretation of experimental practices in modern science are often based on the use of computational modelling” [58]. Despite this, “technology is rarely explicitly called out within definitions of integrated STEM education” [45].

In the problem-solving process, technology can serve one of two distinct roles (c.f., [45]): first, by enhancing efficiency in problem-solving; e.g., by increasing efficiency or collaboration (information technology aspect). Second, by being embedded within the model itself; e.g., because computational concepts like algorithmic thinking are used to create or comprehend parts of the problem (computational aspect). Notably, both roles of technology are dependent on each other [45,65] and they each have benefits that lead to prominent arguments for the inclusion of computing technology for problem-solving and mathematical modeling (e.g., [11,66,67,68]).

Technological support is particularly beneficial for simulations, defined as “the imitation of a real-world process or system over time. Whether done by hand or on a computer, simulation involves the generation of an artificial history of a system and the observation of that artificial history to draw inferences concerning the operating characteristics of the real system” [69].

2.2.4. Model and Result Evaluation

After modeling, an evaluation step is crucial (c.f. [13,70,71,72,73,74]). This evaluation needs to encompass verification and validation activities, whose differences are often overlooked: “there is inconsistency in the meanings of verification and validation in both research and educational literature” [75].

Verification is the process of determining whether the execution of the working process was free from mistakes. Formally defined, verification “refers to the processes and techniques that the model developer uses to assure that his or her model is correct and matches any agreed-upon specifications and assumptions” [76]. Verification activities include checking for mistakes like calculation and implementation errors, comparing values derived from the models with values known to follow from the model [76] (e.g., because an authority like the teacher provided these values for verification), and formally proofing that a proposition does follow from previously stated assumptions [77]. Further techniques are presented in [77,78].

Validation is the process of determining whether a model is applicable for achieving a certain goal in a given situation. It is formally defined as substantiation that a model “within its domain of applicability possesses a satisfactory range of accuracy consistent with the intended application of the model” [79]. Validation techniques include empirical testing to show the correctness of a proposed real-world solution [80], comparing the proposed solution to prior knowledge about the real-world [81], and arguing for the reasonableness of the underlying assumptions of the model [82]. A further technique with high practical relevance is cross-validation: In this approach, two different abstract models are used to represent the same real-world situation. The alignment of predictions from the corresponding parameterized models is then used as an indicator of the validity of both models [83]. Further techniques are presented in [78,84,85].

In problem-solving, the evaluation itself must take into account whether the results are applicable to the problem stated at the beginning. However, evaluation activities should usually also focus on the quality of the model itself [70]. For example, it is desirable to evaluate when the models do and do not work [86].

2.2.5. The SOLO Taxonomy to Measure Structural Complexity

One way to measure the quality of an argumentation is using the structure of observed learning outcomes (SOLO), as described by [87]. This taxonomony introduces five levels that “can be used as at least one important index of structural complexity” of a response. These levels are cumulative, i.e., “[e]ach one adds something to the previous one”.

The lowest of these level is the pre-structural level. An answer at this level adds nothing to the solution. Instead, it may ignore the question (denial), restate given information or the question in different words (tautologizing) or use irrelevant information to answer the question (transduction). The answer “closes without even seeing the problem”.

At the second unistructural level, the answer adds one relevant datum to the answer. There are no further aspects explored. The answer frequently “jumps to conclusions on one aspect, and so can be very inconsistent”.

At the third multistructural level, multiple relevant pieces of information are added. However, they are not weighted against each other or put into a larger context. There is an “isolated fixation on data, and [the answer] can come to different conclusion with [the] same data”.

At the fourth relational level, the different types of relevant information are put into a wider context using relational context information or by being weighted against each other. At this point, the answer usually consistently follows from the data and the answer holds true unless it goes beyond the initially described scenario.

Lastly, at the final extended abstract level, the answer additionally includes abstractions and hypotheses, to make sure the answer is true not only in the current scenario, but also if using different scenarios or edge-cases. The answer consistently follows from the data and clearly states relevant assumptions and limitations.

Additional to these five levels, there are also four intermediary or transitional levels. For example, transitional level 1+ would be a student that “attempts to answer the question but only partially grasps a significant point”.

While these levels were derived based on the cognitive development stages (c.f. [88]), there is no necessary connection between the cognitive development of the person answering and the structural complexity of the answer given. In fact, the SOLO taxonomy was developed precisely because the same persons were observed as giving “a middle concrete response in mathematics [...] followed by a series of concrete generalization responses in geography”. Thus, the cognitive development stage of a person might act as the upper limit to the responses of an participant. For assessment, “the, SOLO levels are equivalent to attainment test results; they describe a particular performance at a particular time”.

3. Simulating Building Evacuations as the Problem for our Study

Based on prior research, we used building evacuation as the domain for our study. As discussed before (see Section 2.1), this domain choice emphasizes two of the STEM subjects: mathematics and technology. Notably, this choice also allows for complex and authentic problems, interesting and authentic activities, and meaningful inclusion of technology, while not being too reliant on domain knowledge or sophisticated mathematical methods (c.f. [89]). Moreover, given the complexity of problems in this domains, digital simulations are frequently used to automate certain steps of the problem-solving process (c.f. [90]).

Two mathematical models are frequently used for such evacuation simulations: grid automata [91] and Flow Networks [92,93].

3.1. Simulating Building Evacuations with Grid Automata

One approach to simulate building evacuations are grid automata. They are based on one of the oldest computing models: the cellular automaton, dating back to von Neumann [94]. Cellular automatons consist of cells with neighbours that change state according to specified rules. Additionally, in a grid automaton, cells are implemented as a 2D grid of rectangular cells.

For simulating evacuations, each cell can be either empty, full, blocked, or safe. Full cells contain (exactly) one agent, empty cells do not. Blocked cells neither do nor can contain an agent. They represent walls. Safe cells remove each agent passing through them from the simulation. They represent the safe destinations.

During each simulation step, each agent can move to a neighbouring cell—either in four (Neumann neighbourhood) or in eight directions (Moore neighbourhood).

Whether or how the agents move is described using the fleeing algorithm. A simple fleeing algorithm might instruct each agent to move to the cell next on the shortest path of unblocked cells to the nearest safe cell if this neighbouring cell is empty.

3.2. Simulating Building Evacuations with Flow Networks

In flow networks, nodes represent positions in 2D-space. They are connected by edges over which agents can move. Each edge has two key properties: delay and parallelism. The delay denotes the time necessary to get from the start of the edge to the destination. The derived speed of the edge is the change in position during the simulated time (frequently provided in pixels per seconds). Parallelism denotes the number of agents that fit on the edge simultaneously.

During simulation, an edge is available, if less agents than the parallelism are currently on it. During the execution of the simulation, each agent uses the available edges to get to one of the safe nodes. The decision about which edge to take and when (or whether to wait for an edge to become available) is described using the fleeing algorithm.

As neighbours can be interpreted as the destination of edges, the same fleeing algorithms can be used for both models. One common fleeing algorithm is the closest goal algorithm: actors are instructed to move to the neighbour that minimizes the distance to the nearest goal, if possible. Further algorithms are described in [95].

3.3. Description of the Problem Used for Our Study

For our study, we needed a specific problem. After considering various options, we selected the task of applying two simulation environments to simulate a scenario and then comparing and evaluating those in regard to the result accuracy. The problem itself was introduced with a motivating text and a visualization of the sports hall as depicted in two distinct simulations, each accompanied by a brief explanation.

The first simulation, referenced in [95] and visualized in Figure 2, operates on a grid automaton. In this setup, every individual occupies a cell measuring 50 × 50 cm² and progresses step-wise to a safe external area. The number of steps taken by the last participant can be used to estimate the evacuation time. The model uses the scale $16\times 16{\mathrm{px}}^2\widehat{=}50\times 50{\mathrm{cm}}^2\widehat{=}1$ cell.

The second simulation, referenced in [89] and depicted in Figure 3, operates on a flow network. Here, each person begins at a node and navigates through edges to reach safety outside the sports hall. The simulation’s duration, influenced by edge delays, can be used to estimate the evacuation time. The model uses the scale $1\mathrm{m}\widehat{=}10\mathrm{px}$. By default, persons are configured to move at 50 px/s.

The exercises used to structure the problem are shown in Figure 4:

3.4. Significance of the Problem

The problem we used is authentic, both in the way that there are professional mathematicians working on this problem (although they may use more sophisticated models, c.f., [96,97,98,99]), and that corresponding activities are typical for approaching STEM problems (c.f. Section 2.1). The problem is also complex, both in regard to the time necessary to solve it, and the number of variables that need to be accounted for (e.g., assumed walking speed, real evacuation time, fleeing algorithm, discretization method).

It also relies on several activities demanded in modern educational settings. For example, the United States’ National Research Council argues that students should “Use (provided) computer simulations or simulations developed with simple simulation tools as a tool for understanding and investigating aspects of a system, particularly those not readily visible to the naked eye” [100]. Furthermore, they should “evaluate and critique competing design solutions” [100] for a problem. Alternatively, the European Commission also lists modelling and simulation as one of the ten key areas of computer science education [101] and a report issued by them highlighted the educational relevance of activities such as the ones used in the task; for example, “computer simulations are often used in science classes to support learning. Learners use simulations to explore phenomena, engaging in what if experiments and reflections while changing the values of the simulation’s parameters” [102].

3.5. The Trap in the Problem

Both simulation environments offered a button to load a pre-implemented sports hall model. However, an intentional inconsistency was embedded: the sports hall within the flow network was scaled threefold. Thus, the built model did not correspond with the actual sports hall representation or a logical parameterized version (i.e., the parameterized and implemented model differed). This deliberate discrepancy aimed to present participants with an identifiable issue during their evaluations in exercises 2–4, without prior warning. Ideally, during their tasks, participants would discern this size disparity in the second exercise, which was designed to promote a side-by-side comparison of the scenarios, particularly spotlighting hallways’ dimensions in the instructions.

One way to identify this size difference is measuring the size of the hallways (visualized in Figure 5). In the grid automaton, there are 6 cells of 50 cm alongside each row of lockers (⇒ 3 m). In the flow network, the edge alongside one row of lockers has a distance of 90 px (⇒ 9 m).

4. Formal Setup

Shortly before the study, the participants received the exercise sheet and links to the two simulation environments. They then had up to four hours to “solve the exercises in a way they themselves would consider satisfactory”. The study itself was conducted with two groups: a student group (6 participants) and a teacher group (14 participants). Initially, the student group was intended to be used as a pilot group. However, as both groups achieved similar results, we decided to merge the results submitted by both into the single dataset we analyze in this paper.

4.1. Prospective Teacher Group

Participants in this group are prospective teachers who are currently studying for a teaching degree for a higher secondary school at our local university. They were advanced in their studies (at least semester six) and are named with an S, followed by a number (e.g., S4). The students received the exercise sheet during a (voluntary) seminar on working with simulations in mathematics education. Two sessions (120 min each) were used, in which the students worked on the exercises on their own in a live session via the video-conference software Zoom. They were allowed to ask questions by (virtually) raising their hand. Questions and answers were discussed in a separate room.

In a feedback round after the sessions, students explained that they understood the exercises but had problems solving them. In addition, two of the students reported that they did not feel comfortable asking questions in this setting.

4.2. Teacher Group

To incentivize asking questions, we changed the setup for the teacher group. This time, the study was conducted in a two-person meeting via Zoom. Participants are named with an P, followed by a number (e.g., P7). At the start of the meeting, the participant was asked to share their screen and explain their working process and thoughts using the thinking aloud method [103]. They were told that they could ask the researcher any kind and any number of questions at any time, that the researcher knows about the inner-workings of both simulation environments, that any question would be answered as briefly as possible in order to not influence their thought process, and that any and any number of follow-up questions were allowed. Very open questions like “What should I do now?” were answered with “I don’t want to influence your approach. What exactly do you want to know?”). Thirteen of the fourteen teachers submitted ahead of time (mostly after 150–210 min of the 240 min working time). One participant (P14) was provided ten extra minutes after the four hours to finish their current thought and was then requested to submit the solution as it was at that time.

The teachers were actively practicing in Bavarian Schools (Southern Germany). Most had a full teaching permit for both mathematics and computer science in Bavaria, Germany. This requires bachelor-level courses in both subjects (including, e.g., courses on software engineering, statistics, analysis, and algorithms), as well as a two year practical training. Two participants only had a mathematics teaching permit, one only had a computer science teaching permit, two further teachers had not yet finished their practical training.

5. Research Questions

The central goal of this exploratory study was the evaluation of educators’ abilities to solve the problem we posed to them and their evaluation of both the model and the results achieved using it. More precisely, we posed four research questions.

First, educators should “experience mathematical problem-solving from the perspective of the problem solver before they can adequately deal with its teaching” [19]. However, there has only been a small amount of prior research on the capability of teachers to solve problems like ours (see Section 2.1.1). Based on this gap, we posed the following question:

Research Question 1:

To what extent were the participants able to correctly estimate the duration with the two simulation environments?

Second, evaluating models and the results achieved with them is a central step while working with models (see Section 2.2.4). However, different goals or kinds of evaluations are often overlooked (see Section 2.2.4). As such, it is necessary that teachers support students by providing and discussing potential and different criteria and that they structure their argumentation by including multiple criteria into an overall assessment. Thereby, “[a] numerical weighting system can help evaluate a design against multiple criteria” [100]. Based on this requirement, we wanted to assess the quality of the argumentation in the evaluation:

Research Question 2:

To what extent were the participants able to argue for or against the realism of their estimated evacuation durations?

Third, being able to solve a problem for oneself does not necessarily enable one to teach well the solving of similar problems. Instead, additional capabilities are required (c.f. Section 2.1.1). As such, we wanted to analyze whether the teachers thought they were sufficiently educated to both solve such problems and teach using them:

Research Question 3:

To what extent do the teachers believe they are sufficiently educated to work on and teach using such problems?

Fourth, it was unclear how good the participants were at judging their own capabilities. Notably, there is little precedence in assessing such a capability. Prior research either focuses on the alignment between students’ self-assessment and teachers’ assessment (e.g., [104,105,106,107]) or with teachers’ self-assessment for problems outside of STEM (e.g., [108,109,110,111]). As such, in our last research question, we wanted to analyze whether there were differences in the self-assessed capability to solve problems like ours and the capabilities shown in the earlier analysis:

Research Question 4:

To what extent was the self-assessment of the participants aligned with the capabilities identified by our assessment?

6. Methods

In general, we used an evaluative qualitative analysis, based on the iterative process of Kuckartz [112], to assess the quality of the evacuation estimates, as well as the argumentation for the evaluation. The main goal of this method is to “assess the data and build categories, whose characteristics are usually noted as ordinal numbers or levels” [112]. Then, a survey using closed questions was used to analyze the self-assessment of the participants. Lastly, the correlation between the answers to the closed question and the assessment levels was used to analyze the dependencies between both. The remainder of this section describes each of the methods in more detail.

6.1. Method for Research Question 1

The evaluative analysis of the solutions was based on eight quality indicators. These were built as follows: Before the evaluation, one of the authors wrote a sample solution for the task. Then, this author proposed quality indicators, read the submitted solutions of the participants, and revised the sample solution and quality indicators. Then, the other authors reviewed the sample solution and indicators and suggested improvements for clarity and completeness. This process was iterated until all authors had agreed that the sample solution and the indicators were suitable for the evaluation.

After creating the indicators, one of the authors classified every solution, to evaluate whether the solutions conformed to the indicator () or did not (). In special cases, a solution could be classified as partly fulfilling an indicator (). For this evaluation, the whole solution was considered: If a participant denoted two different results in exercise 1b (steps and micro-steps) and denoted in exercise 4 that the steps were the correct result to evaluate for realism, then the indicator “exactly one result per simulation is denoted” was fulfilled.

For each indicator fulfilled (out of eight), the participant was awarded one point. For every partially fulfilled indicator, half a point was awarded. The sample solution and quality indicators used are available in Appendix A.

6.2. Method for Research Question 2

The evaluative analysis of the argumentation for the evaluation was based on the levels of the SOLO-taxonomy (see Section 2.2.5). This taxonomy measures “structural organization, which discriminates well-learned from poorly learned material in a way not unlike that in which mature thought is distinguishable from immature thought” [87]. For our purposes, it was superior to the alternative—the Bloom taxonomy [113]—as the latter “is used mostly to set questions and items, not to evaluate open-ended responses to existing questions and item types” [87]. In the evaluation, the solutions were first summarized and paraphrased, then coded as one of the five levels or four intermediary levels (see Section 2.2.5).

6.3. Method for Research Question 3

To analyze the self-assessment of the participants, we asked them to state their level of agreement to the following three propositions using a 5-point-Likert scale:

• I feel sufficiently technically educated to solve such problems (as a learner);
• I feel sufficiently technically educated to teach with such problems;
• I feel sufficiently didactically educated to teach with such problems.

The original questions were provided in German and were provided within five minutes after submission of the solutions. One participant (S6) did not answer the question and had to be excluded from the analysis. Note that “technically” and “didactically” refer to the German phrases “fachlich” and “didaktisch” that do not have a direct English translation. They are used as contrast between education focused on content within a subject (e.g., a mathematics teacher hearing a lecture about calculus or programming) and education focused on educational practices (e.g., a mathematics teacher hearing a lecture about students’ cognitive development or learning theories). Thus, “technically” should not be interpreted to imply technical education as in education in simulation technology).

6.4. Method for Research Question 4

We used correlation analysis to identify potential relationships between the identified variables. Here, we computed both the Pearson (linear correlation, r) and Spearman (rank correlation, $\rho$) correlations between the point score, the SOLO-level (the transitional SOLO-levels 1+ and 2+ were transformed to 1.5 and 2.5 respectively), and the self-assessment of the first two questions. Note that Spearman correlation is more robust and differences between the Pearson and Spearman coefficient can indicate that the correlation is not robust or that the dependency is non-linear.

As our small sample size made analyzing robustness crucial, and we also calculated the skipped correlations [114], i.e., the correlations after removing a data point from the set. Additionally, if applicable, we also calculated the skipped correlation after removing both the highest and lowest scoring participants (according to the point score) from the dataset. Our interpretation of potential dependencies was based on a holistic picture of these correlation values.

In our interpretation, we used the guidelines for behavior science [115] to interpret the strength of any correlation. As such, a correlation of ≥0.1 is considered weak, ≥0.3 medium, and ≥0.5 strong. An example of a medium correlation would be a correlation of r = 0.30 between self-estimated intelligence and the result of a cognitive ability test [116].

Note again that this was an exploratory case study with a limited number of participants, i.e., each dependency found only acted as a basis for a hypothesis that needs to be further tested in a large-scale, quantitative study.

7. Results

In this section, we list the results of the research questions given earlier (c.f. Section 5) and based on the method described above (c.f. Section 6). Note that, for spacial reasons as well as better readability, some details have been moved to Appendix B.

7.1. Research Question 1: Production of Estimates

A full overview of which participant fulfilled which indicator and the mistakes made by individual participants is provided in

Table A1. Evaluation of the solutions according to the indicators. Mistakes made by the participants are listed with a abbreviation in the column mistakes. Evaluations are marked as correct (), incorrect (), or partially correct (). Partially correct solutions are justified in the text.

	E	E1	E2	E3	E4	E5	E6	E7
Participant	One Estimate	Denote Grid Value	Transform Grid	Denote Flow Value	Transform Flow	Consistent Speed	Size Difference	Config Impact	Mistakes	Point Score
S1									R3	6
S2									WT	5
S3										0
S4										8
S5									WT	5
S6									R3, WT	3
P1									WT	4
P2										6
P3									R3	5.5
P4									R3, WT	5
P5										5.5
P6									LW	5
P7									LW	3.5
P8									LW	6
P9									MS, WT	3
P10										6
P11										7
P12										7
P13										3
P14										2
/20	14	18	15	18	7	10	2	7

in Appendix B. There, we also go into more detail about the edge-cases of the coding and discuss every classification classed as partially correct (). Notably, this partially correct classification was given rather leniently.

The distribution of points is shown in Figure 6. Overall, three participants achieved seven or more points, ten participants achieved between five and six (inclusive) points, and seven participants achieved four or less points.

It is noteworthy, that six participants were unaware of the necessary format of the answer—exactly one simulation estimate for each simulation environment—and only half of the participants used consistent assumptions regarding the speed of the agents to calculate their estimates. One aspect only a few participants struggled with was the identification of the correct key result for the simulation environment—for both environments, only two participants did not denote the correct number.

In the end, only four pairs of estimates correctly followed from the simulation environment and only one correctly adjusted for the inconsistent sizes of the sports halls. As such, three quarters of all participants were unable to produce estimates that were both mathematically correct and used consistent assumptions.

Result Summary:

Only four pair of estimates were mathematically correct and used consistent assumptions. Seven participants fulfilled half or less of the criteria. Six participants did not recognize the necessary format of the answer.

7.2. Research Question 2: Evaluation of Realism

From the 20 participants, two did not provide any meaningful information towards an answer. Of the remaining 18, six gave no clear conclusion in regard to whether their estimates were realistic or not. One of them highlighted that their analysis was not sophisticated enough to come to a conclusion but listed aspects that, after evaluation, would allow for a meaningful conclusion. Five argued that their estimates were realistic if certain (stated) assumptions were fulfilled; and two of them argued that these assumptions had been fulfilled. After that, the results of the grid automaton were evaluated to be rather realistic (6 in favor, 3 against); as were the results of the flow network (5 in favor, 4 against).

The main argument used (10 times) was a variation of the following: “If the following assumptions are not fulfilled, the result is not realistic. Otherwise, it is”. For example, P7 argued: “The Model is realistic if the group is guided (e.g., pupils). It is less suited for, e.g., the evacuation simulation of a rock concert” or, less explicitly, P5: “The simulation was programmed to assume the same walking speed for every person. This might not be the case in reality”. Notably, this argumentation never included a justification of why the results would realistic if these assumptions were fulfilled. As such, this line of argumentation uses the absence of evidence as evidence of absence—which is false, as there might be other factors not listed that make the estimates unrealistic. For our evaluation, we coded this as one core argument regarding the realism of the results, even if multiple assumptions were listed within the argument. Only two participants argued for some (but not all) assumptions regarding whether they had been fulfilled, two further participants stated (without reasoning) some (but not all) assumptions, whether they had been fulfilled or not.

The second most used argument was the assertion (without argumentation) that the results did seem realistic or unrealistic (6 times). For example P4: “The results seems rather short to me” or, more explicitly, P9: “on first glance, the result seems to lie within a realistic time-frame”.

The third most used argument was a cross-validation (5 times): since two different models produced results within the same magnitude, both were likely to be realistic.

Three times, a participant argued that a certain assumption had not been (exactly) fulfilled and, as such, the results were unrealistic. For example, P6: “The results of the simulations are unrealistic, because certain aspects that could influence the estimate were not accounted for (see exercise 3). For example, in the simulation, multiple people can be on the same position in space. This is physically impossible” or P1: “for both environments: Is the behavior during fleeing realistic (do really all persons act at the same time?) ⇒ model is good for building a mental concept, but not so much for accuracy”.

Lastly, seven times, the conclusion was—at least in part—based on arguments that were unsuitable for evaluating the realism of the results. This included the usability of the simulation software (2 times), details of their own solution (like specific parameter choices) that could be solved differently (2 times), the comprehensibility of the simulation for students (1 time), the quality of the visualisation (1 time), or properties of the model whose connection to the realism of the results was not explained (1 time).

Regarding the SOLO-levels, we categorized a solution that did not provide an argument as level 1 (2 times); a solution that only used one argument unsuitable for such an evaluation as level 1+ (2 times), a solution that used at least one correct argument as level 2 (6 times); a solution that used the same argument multiple times with a slightly different focus or used additional wrong arguments as level 2+ (5 times), and a solution that used multiple correct lines of arguments to form their conclusion (without connecting them in any way) as level 3 (5 times). A histogram is shown in Figure 7:

Table A2. Overview of the conclusions reached, arguments used, and SOLO-level.

ID	Grid Real.?	Flow Real.?	SOLO-Level	Arguments Used
S1			2+	assertion about magnitude of results; assertion about assumptions
S2	?	?	1	No clear argumentation given (denial)
S3			2+	Listed Assumptions that must be fulfilled; Argued why Assumptions are fulfilled; Visual Representation of the Simulations; Comprehensibility by Students
S4			3	Listed Assumptions that must be fulfilled; Cross-Validation;Usability of Simulation
S5	?	?	2+	assertion about magnitude of results; assertion about assumptions
S6			3	Listed Assumptions that must be fulfilled; assertion about assumptions; assertion about magnitude of result
P1			1+	listed assumptions that are not exactly fulfilled
P2	?	?	2	cross-validation
P3			3	cross-validation; listed assumptions that must be fulfilled
P4			3	assertion about the magnitude of results; listed assumptions that must be fulfilled; details of own solution
P5			2	listed assumptions that must be fulfilled
P6			1+	listed assumptions that are not exactly fulfilled
P7			2	listed assumptions that must be fulfilled
P8	?	?	2	cross-validation
P9	?	?	2+	assertion about magnitude of results; usability of software; listed assumptions that are not exactly fulfilled
P10			3	cross-validation; comparison with real-world event
P11	?	?	2+	listed assumptions that must be fulfilled; analyzed properties of the model; details of own solution
P12	?	?	2	listed assumptions that must be fulfilled
P13	?	?	1	No clear argumentation given (tautologizing)
P14			2	listed assumptions that must be fulfilled

shows an overview of the evaluations and the final assessment of each participant. Notably, not a single participant performed an analysis to quantify the potential impact of a change within the simulation on the overall result, weighted different arguments against each other, demonstrated a relational approach in any way, or made abstractions and generalisations—or even tried but failed in their attempt to do so. As such, no participant was assessed as falling within the level 3+ or higher.

Result Summary:

A significant minority of participants had no final conclusion on whether their estimates were realistic. Five evaluations reached a SOLO level of 3, no argumentation was scored higher.

7.3. Research Question 3: Participants’ Self-Assessment

Most teachers agreed with the proposition “I feel sufficiently technically educated to solve such problems as a learner”: one participant instead answered no ($1\times -1$), three participants had a neutral opinion ($3\times 0$), seven answered rather yes ($7\times 1$), and eight participants answered yes ($8\times 2)$. As such, 15 participants felt sufficiently educated to solve exercises of this type ($\oslash =1.16$, $\sigma =0.90$).

The agreement with the proposition “I feel sufficiently technically educated to teach with such exercises” was lower: four teachers answered rather not ($4\times -1$), five teachers had a neutral opinion ($5\times 0$), seven teachers answered rather yes ($7\times 1$), and three teachers answered yes ($3\times 2$). Thus, ten teachers felt sufficiently educated to teach using exercises of this sort ($\oslash =0.47$, $\sigma =1.02$).

The agreement to the proposition “I feel sufficiently educationally educated to teach with such exercises” was the lowest: five teachers answered rather no ($5\times -1$), six had a neutral opinion ($6\times 0$), five answered rather yes ($5\times 1$), and four yes ($4\times 2$). Thus, nine teachers felt sufficiently educated to teach with exercises of this type ($\oslash =0.42$, $\sigma =1.12$).

Result Summary:

Most participants (15) felt sufficiently technically educated to solve such exercises. Half of them (6) also felt sufficiently (technically and educationally) educated to teach using such exercises

7.4. Research Question 4: Dependency between Self-Assessment and Our Assessment

We provide a full diagram showing the point score, the solo level, and the self-assessment (first question) in Figure 8.

Seven participants either obtained a solo level of 3 (the highest measured amongst our participants) or a result that followed without errors from the simulations; two of these participants obtained both. There was a small to medium correlation between the point scores and the SOLO levels ($r=0.25$, $\rho =0.32$), which was robust if removing any one participant ($0.15\le r\le 0.42$, $0.26\le \rho \le 0.42$), or both the lowest and highest scoring participants ($r=0.32$, $\rho =0.33$).

From the six participants that felt sufficiently educated to both solve and teach using such exercises, two obtained a point score of 5 or higher, and four obtained a point score of 3 or lower. The average point score of these participants was lower ($\oslash =3.66$) than that of the remaining participants ($\oslash =5.25$) Calculating the correlation between the agreement to the first proposition (“I feel sufficiently technically educated to solve such problems (as a learner)”) and the point score resulted in a correlation of $r=0.37$. However, the rank correlation was lower ($\rho =0.17$), indicating that the results might not be robust. And indeed, the skipped correlation was non-existent: Removing the lowest scoring participants led to $r=0.02$, $\rho =0.01$, additionally removing the highest scoring participant made the correlation negative ($r=-0.11$, $\rho =-0.10$). Thus, while there was a medium correlation between the self-assessment of being sufficiently technically educated to solve such tasks and the quality of the solution, this correlation was not robust. As such, we argue that this correlation was an artifact of our small sample size and not indicative of a correlation between the underlying concepts.

The correlation between agreement to the second proposition (“I feel sufficiently technically educated to teach with such problems”) with the point score was negligible ($r=0.13$, $\rho =0.07$). It even becomes negative if one removes the worst scoring participant ($r=-0.11$, $\rho =-0.06$) and stays negative when removing both the best and worst scoring participants ($r=-0.18$, $\rho =-0.13$).

From the six participants that felt sufficiently educated to solve such exercises and teach using them, none achieved SOLO level 3, i.e., took into account more than one suitable argument for the evaluation. The average SOLO level of these participants was lower ($\oslash =1.75$) than that of the remaining participants ($\oslash =2.45$). Calculating the correlation between both variables returned a medium to large correlation, which was negative, i.e., a lower SOLO level correlated with a higher self-assessed ability to solve such exercises ($r=-0.40$, $\rho =-0.42$). This correlation was robust against removing any one participant ($-0.53\le r\le -0.34$, $-0.56\le \rho \le -0.36$), as well as against removing the participants with the highest and lowest point scores ($r=-0.53$, $\rho =-0.53$).

The correlations between agreement to the second proposition (“I feel sufficiently technically educated to teach with such problems”) and the SOLO levels was also negative and of a similar magnitude, even if a little smaller ($-0.44\le r\le -0.28$, $0.43\le \rho \le -0.26$).

Result Summary:

The quality of the simulation estimates and the quality of the evaluation were at least weakly correlated. The quality of the simulation estimates was likely uncorrelated with the self-assessment. The quality of the evaluation with the self-assessment had a negative correlation that was of at least medium strength.

8. Interpretation

This exploratory research had the primary goal of assessing the abilities of educators to work with models and evaluate models and their results. To do so, we assessed the quality of twenty educators—six students studying for a teaching degree and fourteen practicing teachers. In this section, we try to generalize from the patterns in the results we observed.

8.1. Lack of Competence in the Assessed Activities

The first key result is that only a small minority of our participants produced a result that correctly followed from the provided model and technology. Furthermore, in the evaluation, only a small minority of participants used multiple distinct arguments in their conclusion and no one weighted them against each other or used other techniques to combine them into a full picture (see Section 7.1 and Section 7.2).

This is concerning, as such activities are indeed part of many curricula (see Section 3.4). As such, there might be a gap between the demands of modern educational research and curricula and the capabilities of current educators.

Notably, the extent of this gap cannot be assessed using our work. For such an assessment, a qualitative study with a larger sample size must be used. However, there is a sharp contrast between the frequent and confident implication that the vast majority of educators are able to do everything written in a curriculum, on the one hand, and the assessed capabilities in this exploratory study, on the other hand. Based on our results, it might be worthwhile to actually conduct such a large-scale study, to assess the existing capabilities of educators with regard to complex and authentic problems as demanded by both curricula and research.

8.2. Gap between Self-Assessment and External Assessment

The second main result is the gap between the self-assessment of teachers and the external assessment in our study. For the first activity assessed, the self-assessment was uncorrelated with our assessment. For the second activity, there was a strong negative correlation. As such, at least some participants were unaware of their own limitations [117].

This is a problem for any situation that explicitly or implicitly relies on such an self-assessment. An example would be voluntary training for practicing teachers with regard to evaluation problems. Given the negative correlation, such a training might be predominantly chosen by teachers that are aware of their own limitations and recognize the problem as complex, but not by those that are unaware of their limitations and think their current solution is adequate. As such, such a voluntary training might not be attended by those who have the highest requirement for it.

8.3. Independence of Sub-Skills of Problem-Solving

A third result of our exploratory study is that it might be more suitable to describe problem-solving as a composite of various sub-skills, rather than one atomic skill. While it is likely that there are multiple weak or medium-strong correlations between several sub-skills of complex and authentic problem-solving (or with general intelligence), our data do not suggest that these connections are strong or easily spotted when using a small sample size. Thus, instead of speaking of the capability to solve complex problems, it might be more suitable to develop a competence model based on different sub-skills of this process.

Similarly, it might be useful if educational interventions were described with sufficient details to identify the sub-skills of problem-solving that are fostered. This might also increase the relevance of problem-solving overall: If the multiple distinct sub-skills necessary for problem-solving were identified, it would be possible to include time for focusing on each of them in curricula—rather than focusing on problem-solving as singular entity. To this end, developing a classification system to describe which kinds of problems are suitable for teaching which kinds of sub-skills might be more helpful.

9. Limitations

In this section, we address potential limitations in our study method and their subsequent implications.

9.1. Sample Characteristics and Size

This study reports on results of an exploratory study with $n=20$ participants. This size limitation is mostly attributed to difficulties in recruiting practicing teachers for such a long setup. While we consider this sample size appropriate for an exploratory study to formulate initial hypotheses, further quantitative confirmation is necessary before greater emphasis can be put on our generalizations in Section 8.

Similarly, our participants were largely from the same educational background in Bavaria, Germany—receiving similar education, primarily focusing on in-depth knowledge in mathematics and computer science but not on applied problem-solving. Our participants differed in their school type (2 × Elementary Education, “Grundschule”; 5 × lower secondary education, “Realschule”; 13 × higher secondary education, “Gymnasium”), age (20–60), and gender (13 × female, 7 × male). However, it remains uncertain how these findings would apply to teachers from different regions or backgrounds.

9.2. Task Validity and Specificity

The appropriateness of the task we selected to assess teachers’ abilities remains in question. Our task design aimed to mirror typical school tasks (see Section 2.1 and Section 3.4) in modern settings; however, its effectiveness in doing so has not been confirmed, given the absence of measurement tools assessing problem similarity or complexity. To ensure a comprehensive understanding, future studies should incorporate problems from various domains, of various types, focusing on diverse aspects of problem-solving, and spanning multiple STEM subjects.

Furthermore, the specifics of the setup we used (like the specific simulation environments) could and should also be varied. While we tried to mitigate problems by being lenient in our assessment, the exact impact of specific implementation choices are currently unclear.

9.3. Influence from the Setup

We summarized results from both student and teacher groups, due to their similar outcomes. Though the overall findings remained consistent, some nuances existed when considering each group individually. This merging was notable, especially since the groups had different levels of supervision and question-asking tendencies (students asked a total of 2 questions, the teacher group 237).

However, one has to note that these questions were not used as well as possible by many participants: In the person-to-person meetings, participant asked an average of $19.75$ questions (range: 6–35). This indicates that the barrier to asking question had been sufficiently reduced. However, analysis of these questions shows that a large percentage of them (≈40%) focused on program usage (e.g., “which button do I need to press to load the sports hall?”). Questions regarding the model itself (e.g., “can you explain the variable waiting times?”) were far less frequent (≈12%). As such, it is likely that the answers frequently only sped up the working process, rather than leading to additional insights into the model or problem-solving process.

9.4. Focus on the Written Solutions

Our analysis focused on the written solutions of the teachers. Thus, it might be possible that the teachers used far more sophisticated lines of argumentation and reasoning to come to their conclusion than they cared to write down. While possible, we consider this to be very unlikely, as an author was present during the whole solving process and the teachers, asked to verbally discuss their thought process, did not show clear indications of writing down significantly less arguments or weighting between arguments than they thought of.

10. Conclusions

This research focused on a case study based on simulating building evacuations to analyze the problem-solving capabilities of educators in the context of problem-centric education. In our study, we performed two evaluative analyses into the solutions of twenty educators asked to evaluate the realism of two evacuation estimates. Three key findings were identified.

First, aligned with existing research (see Section 2.1.1), a significant number of teachers struggled when working on the problem. This was likely not due to the mathematics involved, but because of difficulties in applying mathematics to real-world contexts. For example, errors in the application of the rule of three were significantly less common than the usage of inconsistent assumptions regarding the walking speed when creating the duration estimates. Additionally, the argumentation used for the evaluation was rather superficial. Techniques like analyzing the impact of a potential change to the estimates or weighting different arguments against each other were not used.

Second, the self-assessment of teachers was either uncorrelated or negatively correlated with the assessment we conducted. This was especially true for the correlation between the self-assessment as being able to solve exercises like the one we provided and the ability to use sophisticated argumentation.

Third, the weak correlation between the different sub-activities necessary for problem-solving strengthens the position (c.f. [25]) of problem solving as a multifaceted combination of skills, each of which do not necessarily correlate strongly with one another.

The study’s main limitations included a limited sample size of 20 participants, who were predominantly from a similar educational background in Bavaria, Germany. Furthermore, the validity of the task designed to assess the teachers’ abilities is uncertain, especially with no established tools available for gauging its similarity or complexity compared to other educational tasks. Although the results from the student and teacher groups were combined due to similar outcomes, there were notable differences in their interaction levels and the types of questions asked, emphasizing software use over underlying model understanding. Lastly, while the analysis prioritized written solutions, it is believed that these sufficiently represented the teachers’ thought processes, given the consistent observation during their problem-solving sessions.

Based on this exploratory study, two main lines of future work emerge. First, it might be useful to assess the capabilities of teachers to solve problems as used in modern education styles in more detail, especially based on typical examples from the local curriculum. A large-scale, representative, quantitative analysis might reveal and quantify previously unseen gaps that then could be addressed through additional teacher training. Second, it might be worth investigating the reason for this gap further. This could include both a more detailed analysis of the approach taken by teachers to such exercises and identifying which steps exactly (like coming up with potential arguments, applying arguments, weighting arguments) teachers struggle with. Furthermore, it might also be worthwhile analyzing whether the tendency to focus on larger-scale problems in education and on problem-solving activities (rather than content knowledge) is also reflected in current teacher education.

Overall, working on complex and authentic problems is seen as more and more relevant in STEM education. However, our findings hint at a potential need for more nuanced teacher training, emphasizing the translation of math to real-world situations, better argumentation based on mathematical models, and a deeper understanding of the composite sub-skills involved in problem-solving.