The System Architecture-Function-Outcome Framework for Fostering and Assessing Systems Thinking in First-Year STEM Education and Its Potential Applications in Case-Based Learning

Abstract

Systems thinking is crucial for understanding and solving complex problems and is considered an important thinking skill in engineering. Active learning is considered an effective approach for fostering STEM students’ systems thinking. However, viable methods for teaching and assessing systems thinking with active learning across STEM disciplines, particularly in first-year undergraduate education, are still under-researched. In this paper, we introduce a research-based framework named System Architecture-Function-Outcome to help first-year STEM instructors both foster and assess students’ introductory systems thinking. To conduct an initial evaluation of the framework’s suitability in active learning settings, we designed a directed case-based learning assignment with an adapted article and a rubric for assessing ‘introductory systems thinking’, as defined in the framework. We deployed the assignment among 84 first-year STEM students and successfully tested its inter-rater reliability, with 75–100% inter-rater agreement across all assessment criteria. We discuss the implications of our results on fostering and assessing first-year STEM students’ systems thinking, and outline examples for potential applications of the framework, pending further validation, in case-based learning settings of varying degrees of learner autonomy, from lecture-based to problem-based learning.

1. Introduction

In this work, we present an updated version of the System Architecture-Function-Outcome (SAFO) pedagogical framework for systems thinking about technological systems. We detail how we implemented the framework in a case-based learning (CBL) setting and outline the potential applications of this framework in various CBL settings with varying learner autonomy. As we discuss in Section 1.2, the uniqueness of the SAFO framework is that it is the only pedagogical framework, as far as we are aware, that is specifically tailored to tackle the challenges involved in both fostering and assessing the systems thinking of first-year STEM students, i.e., the introductory systems thinking (IST) of novices. The work described in this paper is a first step toward valid and a reliable pedagogy to foster and assess systems thinking in first-year STEM CBL.

In this section, we discuss the need and challenges in fostering and assessing systems thinking in first-year STEM education, and describe an existing pedagogical framework developed to enable this effort, the SAFO framework, which is meant to teach IST to STEM students. We describe case-based learning (CBL) and discuss the pros and cons of different variations of CBL to fostering systems thinking. We end this section with the main aims and conclusions of our study.

1.1. Fostering and Assessing Systems Thinking in First-Year STEM Education

1.1.1. What Is Systems Thinking, and Why Foster It in First-Year Students?

The definition of ‘system’ is subject to rich and ongoing academic discourse [1]. With our STEM education and systems focus and based on previous work on the framework presented herein [1,2,3,4], our working definition of a technological system is as follows: a technological system is an entity composed of interacting parts, delivering a predetermined function through its architecture—the latter being a combination of its structure and behavior. The system’s function is delivered through interactions between parts of the system, both internally and with the system’s environment. All these interactions are cause-and-effect relationships. Some whole-system properties are different to those of the system’s individual parts. The system’s purpose is to provide a predetermined benefit to group(s), which are the system’s beneficiaries. The system’s purpose is delivered by its function. Examples of technological systems included under this definition can include a bicycle, a coal-fired power plant, or an e-reader, among numerous others.

The concept of systems thinking has been independently developed in various disciplines, such as engineering [2], natural science [5], and social science [6]. Systems thinking in natural science can be conceived of as describing or analyzing natural phenomena, events, or circumstances—physical, chemical, biological, or some combination thereof—as if they were systems [7]. In the context of technological, engineered systems, systems thinking can be considered as a holistic understanding of the system’s function, structure, and behavior, and how the latter two interact to deliver the former [1,2]. Systems thinking in STEM can be thought of as a higher-order thinking skill, or set of skills, enabling one to identify, understand, predict, and suggest improvements for every aspect of a technological system, and for the way these aspects inter-relate within the system [1,2,8]. Creating a common language for systems thinking across science and engineering disciplines is a subject of ongoing scholarly inquiry and debate [9]. In this paper, we do not aim to engage in that debate but instead build on prior work concerning systems thinking assessment in first-year STEM education [3].

Applications of systems thinking exist in the teaching of many STEM disciplines [10]. The US National Research Council [11] presented a framework for 21st century skills in science and engineering education—primary, secondary, and higher—which included systems thinking. Additionally, the ABET 2022–2023 criterion 3 (student outcomes) for accrediting engineering programs [12] includes the ability to produce engineering solutions while considering factors that require a systems perspective, namely global, cultural, social, environmental, and economic factors. Systems thinking is also part of the European Sustainability framework [13] and the K-12 Standards for Technological and Engineering Literacy of the International Technology and Engineering Educators Association [14].

1.1.2. How Can Systems Thinking Be Fostered and Assessed in First-Year STEM Students?

If fostering systems thinking is important in undergraduate engineering education, then, we argue, engineering schools should begin teaching it in the first year. However, teaching systems thinking in first-year engineering education is different to teaching it in later years, because first-year students are not yet specialized in any sub-discipline of engineering and have at best a rudimentary understanding of technological systems from an engineer’s point of view [15]. Therefore, a pedagogical framework for teaching IST in first-year engineering education must be rooted in the fundamentals of engineering while also be agnostic to any sub-discipline of engineering.

Teaching systems thinking in first-year STEM education presents additional challenges to instructors. As ref. [3] showed, there is a lack of frameworks that enable instructors in undergraduate engineering education to both foster and assess systems thinking or design multiple types of systems thinking assignments (and assessments).

In their systematic review of the literature on systems thinking assessment in engineering, Dugan et al. [16] identified 27 unique assessments. They derived and defined four types of assessments, namely behavior-based, preference-based, self-reported, and cognitive activation, and eight assessment formats, including mapping, scenario, open-ended, and oral, among others. They highlighted a lack in assessments which account for both technical and contextual considerations of systems thinking.

The SAFO framework is an updated version of the System Architecture-Function-Purpose (SAFP) framework, which is behavior-based, examining knowledge and skills, and is ‘open-ended’ in format, allowing students to describe the system using free text. The framework was developed to teach first-year STEM students about (mostly) technological systems in a discipline-agnostic way [3] and draws in part on previous work on an assessment method which made use of conceptual models of systems created by students to assess their systems thinking [17]. In this way, the SAFO framework’s purpose is to help instructors both foster and assess STEM students’ IST. The framework allows for the inclusion and distinction between technical and contextual aspects of a system and is simple enough to be implemented without prior expertise or use of special software.

The changes made to SAFP in the SAFO framework were as follows: ‘Function’ was split into ‘Input’ and ‘Output’; ‘Detriment’ was added; and ‘Purpose’ was renamed ’Outcome’ and now also includes ‘Detriment’. These changes were made to improve the construct’s ability to describe technological systems more comprehensively. In particular, the addition of ‘Detriment’ as a new element under ‘Outcome’ extends IST to include not only the (intended) positive effects of the system on people, but also its expected negative effects on people, helping students see beyond the narrow view of a technological system as something which is purely beneficial and adding a layer of complexity to their understanding of these systems.

Table 1. The System Architecture-Function-Outcome framework construct: terms and descriptions.

System Aspect	System Element	Description
Architecture	Structure	Key parts of the system of interest (SoI).
	Behavior	Cause-and-effect interactions between the key parts of the SoI.
Function	Input	A system that exists on the boundary of the SoI and which provides it with input.
	Output	A system that exists on the boundary of the SoI and which receives output from it.
Outcome	Stakeholders	Groups of people most affected by the problems which the SoI is designed to solve.
	Problem	Key problems faced by the key stakeholders.
	Benefit	Key positive intended outcomes of the SoI on the key stakeholders.
	Detriment	Key negative expected outcomes of the SoI on the key stakeholders.

shows the SAFO construct, and

Table 2. The System Architecture-Function-Outcome framework construct applied to a petrol-powered, manually driven car as the system of interest.

System Aspect	System Element	Description
Architecture	Structure	Steering wheel; gear box; engine; wheel axle
	Behavior	Gear box transmitting to engine; engine spinning wheel axle; steering wheel rotating wheel axle
Function	Input	Human driver commanding manually driven petrol-powered private car
	Output	Human driver commanding manually driven petrol-powered private car
Outcome	Stakeholders	Working adults who live far away from their workplace
	Problem	Human driver commanding manually driven petrol-powered private car
	Benefit	Commuting to work quickly, conveniently, and reliably
	Detriment	Increases air pollution within cities, which leads to adverse effects on city dwellers’ health

shows an example of mapping a technological system—a manually driven, petrol-powered car—to this construct. The SAFO construct is not meant to be a fundamental or comprehensive construct of systems thinking. Rather, it was developed by the first author to represent aspects of systems thinking that we believe should be included when introducing systems thinking about technological systems to STEM novices.

Table 2 shows the SAFO construct applied to a petrol-powered car as the system of interest (SoI).

Regarding the ‘Architecture’ and ‘Function’ aspects, it should be noted that every technological system can be a part (sub-system) of another, bigger system architecture, and/or comprised of sub-systems with their own architectures. In this way, the ‘Function’ of a system can itself be described as the ‘Architecture’ of a bigger SoI. For example, the gear box included under ‘Structure’ in Table 2 can be described as an SoI with its own architecture, while the car can be described as one part of a larger architecture of a city’s transportation system.

1.2. Case-Based Learning

1.2.1. What Is Case-Based Learning?

CBL is a form of active learning. This typically involves (a) learners applying knowledge and higher-order thinking to problems, cases, scenarios, etc., (b) working in groups, and (c) reflecting on their own learning [18,19]. Systems thinking itself is a form of higher-order thinking [20], and active learning in STEM education has been shown to help foster students’ higher-order thinking skills, e.g., [18,19,21,22,23,24,25]. We argue that CBL can provide opportunities for students to apply and develop their systems thinking.

1.2.2. Case-Based Learning Types

The origins of CBL are in professional education, specifically in medicine, business, and law. CBL is a name for many pedagogical approaches which use specific occasions (cases), thus contextualizing the learning of a discipline and/or practice-specific knowledge. CBL can help facilitate students’ development of conceptual understanding along with their thinking skills, as students work through and reflect on the process of solving cases [26].

As a field with an established tradition in CBL, medical education can offer us evidence-based ways for classifying CBL implementations. Kulak and Newton [27] classified CBL implementations in medical education into five types, varying by the degree of the learner’s direction over their learning (learner autonomy).

Table 3. Types of case-based learning implementations and their respective learning progressions. Adapted from [13].

Case-Based Learning Type Learner Autonomy Level	Teaching and Learning Activities
Lecture-based Very low autonomy	Instructor gives lecture which includes a description of the case; Students take knowledge quiz/test.
Directed Low autonomy	Instructor gives lecture which includes a description of the case; Instructor provides closed-ended questions/well-structured problems to students; Students answer questions/propose solutions; Instructor moderates classroom discussion about questions and answers.
Interrupted Medium autonomy	Instructor provides students with partial details of case and with closed-ended questions/well-structured problems; Students answer questions/propose solutions; Instructor provides students with more details about case; Students integrate new information into their answers/solutions.
Jigsaw High autonomy	Case is provided to multiple groups. Each group receives only one of multiple learning topics, phrased as questions; Each group solves their question; New groups are formed with a representative from each of the previous groups, so each member shares their answers to the questions; The group integrates all answers into a general solution to the case.
Problem-based learning Very high autonomy	Depending on the learning goals, a full or briefer version of the case is given out. No questions are provided to any group, and there is no lecture; Each group develops their own questions. Learning issues may go beyond the stated topic but remain related to it.

summarizes these types of CBL in general terms that can also apply to fields outside of medical education.

Lavi and Marti [28] reviewed the literature on CBL in undergraduate engineering education and found four ways in which cases were used in empirical studies: (a) embedded—one or more cases embedded throughout course or module; (b) multi-standalone—one or more standalone cases implemented in multiple courses; (c) single standalone—one or more standalone cases, each implemented in one course; and (d) assessment—standalone case(s) used for summative learning assessments in course.

General challenges for implementing CBL include a lack of instructor training, challenges in choosing suitable assessment methods, and the need to train students in these non-traditional learning methods, which often stand in the way of implementing CBL. Also, studies on CBL are under-published in undergraduate engineering education, so educators have little insight to draw from [29].

1.2.3. Choosing between Case-Based Learning Types

PBL is often associated with engineering education, as engineers are typically tasked with solving problems [30]. While PBL settings help mimic authentic problems that students will experience after their studies, implementing PBL can naturally also be time-consuming and costly [31]. Compared to PBL, in which problems (or cases) are open-ended, ill-defined, and at times even student-led, CBL types other than PBL afford students less autonomy (see Table 3). In CBL types like lecture-based and directed CBL, problems (or cases) are often defined to an extent and the problem-solving process is guided to one degree or another [32].

PBL leaves more of the preparation stage of problem-solving to students than other types of CBL do. When it comes to teaching and assessing specific skills such as systems thinking, non-PBL CBL can help instructors scaffold students’ learning more so than PBL, thus ensuring that certain learning outcomes are met. Indeed, Lavi et al. [2] found that studies in undergraduate engineering education focused on fostering students’ systems thinking all included CBL assignments which afforded students less autonomy and were not PBL. This finding may point to non-PBL CBL types as being more suitable for the very first introduction of systems thinking to first-year students. A detriment, on the other hand, is that problems might not be presented in a way that is as authentic.

A potential response to the conundrum presented above of which CBL type to implement with students might be to implement a progression of CBL types across the learner autonomy spectrum, starting with forms of CBL that afford students less autonomy, like lecture-based or directed CBL, and ending with jigsaw CBL or PBL. In this study, which focuses on directed CBL, we show how such a learning progression could potentially start.

1.3. Main Aims and Principal Conclusions

This study has two aims: (a) to develop a research-based instrument, an assessment rubric, for introductory systems thinking based on the SAFO framework, and evaluate its inter-rater reliability; and (b) provide examples for applying this framework in various CBL settings for the purpose of fostering, assessing, and researching first-year STEM students’ introductory systems thinking.

The study highlights the framework’s suitability for directed CBL settings. We successfully developed and validated this assessment rubric based on the SAFO framework, demonstrating high inter-rater reliability. This rubric allows for clear and detailed assessment of students’ introductory systems thinking.

2. The Introductory Systems Thinking Assessment Rubric: Instrument Development and Deployment

2.1. Context and Participants

2.1.1. Learning Setting

The case was integrated into a first semester PBL course, which is a mandatory course offered to all students across the science and engineering departments at Aalborg University. It is organized in two parts: one focusing on collaborative learning and team skills (process perspective), and one focusing on problem analysis and complex problem-solving skills (problem perspective), in which students learn about problem analysis, the methodology and theory of science, stakeholder engagement, and sustainability. In this study, the systems thinking article assignment was integrated into the problem perspective in the PBL course at the Department of Biochemistry, with students from the departments of Biology, Biotechnology, Chemistry, Chemical Engineering, and Environmental Science participating in the study.

While the study was conducted as part of a PBL course, the study assignment itself would not fall under PBL in the CBL typology (see Table 3), as it does not present students with an ill-defined or open-ended problem, but instead it presents students with a case that is somewhat defined within a case description (short article) that is referred to as the authoritative source of ‘truth’ about the system. This makes this study’s learning setting more suitably called ‘Directed CBL’, which is the CBL type that affords the second-least amount of autonomy to learners (see Table 3). The selection of this type of CBL was in line with the validatory goal of developing an assessment rubric based on the SAFO framework, as we wanted to reduce the variability of the learning setting (and study conditions) to what we considered to be an optimal minimum. We did not select the even lower learner autonomy ‘Lecture CBL’, as this type of CBL allows for very little variance in students’ responses. Once the assessment rubric is validated, the SAFO framework can be applied to other parts of the PBL course and used to design and evaluate study assignments in line with other variations of CBL, including student semester projects.

2.1.2. Participants

A total of 123 students gave their agreement to participate in the study. Out of those 123 students, 84 filled out both the background survey and article assignment forms in full. We analyzed the responses from those 84 submissions.

Table 4. Participant details.

Variable	Value	N
Gender	Woman	42
	Man	41
	Non-binary	1
Prior education	General high school	49
	Technical high school	27
	Other	8
Current Discipline	Biology	21
	Biotechnology	24
	Chemistry and Chemical Engineering	24
	Environmental Science	15

summarizes the demographic and educational details of the participants of the study.

2.2. Data Collection

2.2.1. Procedure

Data collection included three online forms: informed consent to participate in the study, a background survey, and an article assignment.

The morning of the day the data collection took place, the second author described the SAFO construct and the study description to the students, followed by the deployment of the informed consent form to the students. The first author then joined remotely and took 10 min to introduce the SAFO construct, which included an applied example of a bicycle, a relatively simple system. Students were then given 10 min to fill-out and submit the background survey, followed by 40 min to fill out and submit the article assignment form individually. The students’ responses to the article assignment were then analyzed based on an assessment rubric developed by both authors.

2.2.2. Background Questionnaire

This questionnaire included items pertaining to the respondents’ demographic and education details.

2.2.3. Article Assignment

The topic of the article—pen-farming of salmon—was already included in the course curriculum as a sustainability case and overall theme in which the students were to identify and define a problem related to their specific discipline over the course of two weeks. The article was written based on [33,34,35] and reviewed twice by an agricultural engineer. See Appendix A for the article.

Table 5. Instructions for article assignment.

System Aspect	System Element	Prompt
Architecture	Structure	Name five key parts of the system.
	Behavior	Name four causal interactions between the parts you mentioned in the previous question. An interaction should be between two or more parts.
Function	Input	Describe a system that exists on the boundary of our system of interest and which provides our system with input. What is this ‘input system’? What is its input into our system of interest?
	Output	Describe a system that exists on the boundary of our system of interest and which receives output from our system. What is this ‘output system’? What is the output it receives?
Outcome	Stakeholders	System outcome-key stakeholders: what group of people is most affected by the problem which the system function solves or improves?
	Problem	What is the key problem the system is designed to solve for its key stakeholders?
	Benefit	Describe a key positive intended outcome of the system when it functions as intended. The outcome should affect the key stakeholders of the system.
	Detriment	Describe a key negative expected outcome of the system when it functions as intended. The outcome should affect the key stakeholders of the system.

shows the instructions for this assignment.

2.3. Assessment Rubric Instrument

The assessment rubric instrument was developed by both authors based on the ST construct. The rubric includes two criteria, ‘adherence’ and ‘correctness’. Adherence relates to whether the students’ responses demonstrated an understanding of the SAFO construct, as described in Table 1, and how closely the assignment instructions were followed. Correctness relates to the content of the responses in relation to the source of truth, which was the short article (case description). We treated adherence and correctness as separate criteria because understanding the SAFO construct did not necessarily imply an understanding of the short article’s content. Having a high level of IST requires both an understanding of the SAFO construct and an understanding of the short article’s content.

Each criterion was scored as 0, 1, or 2, across all the elements of the SAFO framework, and the IST score for each system element was calculated as [adherence score] ∗ [correctness score].

Table 6. Matrix of scores for introductory systems thinking.

Score		Adherence to Instructions
		0	1	2
Correctness of response	0	0	0	0
	1	0	1	2
	2	0	2	4

shows the range of possible scores for each system element. For the assessment rubric, including detailed scoring guidelines, see Tables S1 and S2 in the Supplementary Materials.

The initial version of the assessment rubric (scoring guidelines) provided detailed instructions for how to score ‘adherence’ and for how to score ‘correctness’ for every system element. The guidelines for ‘adherence’ were refined using three randomly selected samples from the students’ submitted assignment forms. The authors scored these samples separately according to the scoring guidelines for ‘adherence’. Framework adherence is important as systems thinking is not about having a comprehensive overview of every single part of a system or all its boundary systems, but rather being able to identify essential aspects of a system and understand their function and important interactions on their boundaries. On comparing their scores, the authors found some differences, which led to modifications and elaborations in the scoring guidelines. This process of separate scoring, comparison of scores, and changes to the rubric were repeated twice more, until both authors agreed that the scoring guidelines were ready for inter-rater agreement testing.

Next, the first author randomly selected eight samples, comprising 10% of the total number of samples (N = 84), and sent these to the second author. These samples were then also scored separately by the two authors, and the percentage of agreement on the scores was calculated to determine the inter-rater agreement for ‘adherence’.

Finally, the scoring guidelines for ‘correctness’ were refined. The same process was repeated here as for ‘adherence’, with the only exception being that this time the authors agreed to go for inter-rater agreement testing after four iterations like the ones described above for scoring ‘adherence’.

Table 7. Scoring guidelines for the ‘adherence’ criterion of introductory systems thinking.

System Aspect and Element 1		2	1
Architecture	Structure	Five different parts AND majority are technological	Five different parts AND majority are not technological OR Four or six different parts AND majority are technological
	Behavior	Four different explicit interactions (cause-and-effect relationships) that together cover every part mentioned under ‘Structure’	Four different explicit interactions that together do not cover every part mentioned under ‘Structure’ OR Three or five different explicit interactions that together cover every part mentioned under ‘Structure’ OR Four different implicit interactions that together cover every part mentioned under ‘Structure’
Function	Input	One Boundary System AND one interaction between Target System and Boundary System. The direction of the interaction is from Boundary System to Target System. Input and output systems can be the same.	One Boundary System AND multiple interactions between Target System and Boundary System. The direction of all the interactions is from Boundary System to Target System. OR One non-Boundary System AND one interaction between Target System and non-Boundary System. The direction of the interaction is from non-Boundary System to Target System.
	Output	One Boundary System AND one interaction between Target System and Boundary System. The direction of the interaction is from Target system to Boundary System. Input and output systems can be the same.	One Boundary System AND multiple interactions between Target System and Boundary System. The direction of all the interactions is from Target system to Boundary System. OR One non-Boundary System AND one interaction between Target System and non-Boundary System. The direction of the interaction is from Target System to non-Boundary System.
Outcome	Stakeholders	One distinct group	Multiple distinct groups
	Problem	One problem/need/lack/demand of stakeholders	Multiple problems/needs/lacks/demands of stakeholders
	Benefit	One direct positive outcome affecting stakeholders, with or without a causal explanation.	Multiple direct positive outcomes affecting stakeholders, with/without a causal explanation. OR One indirect positive outcome affecting stakeholders, with or without a causal explanation.
	Detriment	One direct negative outcome affecting stakeholders, with or without a causal explanation.	Multiple direct negative outcomes affecting stakeholders with/without a causal explanation. OR One indirect negative outcome affecting stakeholders, with or without a causal explanation.

¹ ‘Interaction’ must be a causal relation, not a structural one. Interactions can involve matter, energy, or information. Behavioral interactions are time-dependent and dynamic.

and

Table 8. Scoring guidelines for the ‘correctness’ criterion of introductory systems thinking.

System Aspect and Element 1		2	1
Architecture	Structure	All parts are correct and precise	Half or more of the parts are correct and precise, but not all
	Behavior	All interactions are correct and precise	Half or more of the interactions are correct and precise, but not all (ignore non-behavioral relationships)
Function	Input	Input system and its interaction with the system of interest (SoI) are both correct and precise	Input system is correct but imprecise/general/vague If multiple systems: half or more of the input system is correct and precise
	Output	Output system and its interaction with the SoI are both correct and precise	Output system is correct but imprecise/general/vague If multiple systems: half or more of output systems is correct and precise
Outcome	Stakeholders	All groups are correct and precise	If multiple groups: half or more of the groups are correct and precise, but not all If one group: correct but imprecise/general/vague
	Problem	All problems are correct and precise	If multiple groups: half or more of the problems are correct and precise, but not all If one problem: correct but imprecise/general/vague
	Benefit	All benefits are correct and precise	If multiple benefits: half or more of benefits are correct and precise, but not all If one benefit: correct but imprecise/general/vague
	Detriment	All detriments are correct and precise	If multiple detriments: half or more of detriments are correct and precise, but not all If one detriment: correct but imprecise/general/vague

¹ Responses that received ‘0’ for adherence automatically receive ‘0’ for correctness. Ignore parts of responses that did not adhere to instructions. Each aspect should be scored independently of all other aspects. The correctness of a response should be based on the sources of truth provided to students, including reasonable responses they are likely to make based on the content of those courses.

show the final guidelines for scoring the ‘adherence’ and ‘correctness’ criteria, respectively, and which we used to establish the rubric’s inter-rater reliability.

3. The Introductory Systems Thinking Assessment Rubric: Data Analysis and Results

3.1. Data Analysis

Background survey: Analysis included only descriptive statistics.

Article assignment–inter-rater reliability for adherence and for correctness: We calculated the inter-rater reliability through the percentage of agreement on scores between the two raters.

Article assignment scoring for introductory system thinking: Once the inter-rater reliability was established for the assessment rubric, the first author scored the entire dataset of 84 responses for adherence and correctness and calculated individual IST scores.

Table 9. Median scores for adherence, correctness, and introductory systems thinking.

	Median Score
System Element	Adherence	Correctness	Introductory Systems Thinking
Structure	1.00	1.00	1.00
Behavior	1.00	1.00	1.00
Input	1.00	1.00	1.00
Output	1.00	0.50	0
Stakeholders	1.00	1.00	1.00
Problem	1.00	0	0
Benefit	2.00	1.00	1.00
Detriment	1.00	1.00	1.00

shows the median scores for adherence, correctness, and IST. Every element featured the full range of IST scores (0, 1, 2, and 4).

3.2. Inter-Rater Reliability

The two raters had above 75% agreement on scores for every system element.

Table 10. Inter-rater agreement for adherence and for correctness.

System Aspect	System Element	% Inter-Rater Agreement on Scores
		Adherence (N = 8)	Correctness (N = 8)
Architecture	Structure	100	88
	Behavior	75	88
Function	Input	88	100
	Output	88	88
Outcome	Stakeholders	88	100
	Problem	100	100
	Benefit	88	75
	Detriment	88	100

shows the percentage of agreement between the raters for adherence and for correctness.

We also tested whether the students’ perceptions of the sufficiency of the time allocated for the assignment (1—not enough time; 2; or 3—more than enough time) and the article’s clarity (1—‘very unclear’; 2; 3; 4; or 5—‘very clear’) affected their IST scores. Independent-sample Kruskal–Wallis tests found no significant differences based on the time scores or based on the clarity scores for any of the system elements.

4. Discussion

In this section, we discuss our descriptive findings, the potential use of the instrument for formative assessments, how the instrument can be used for prescribing systems, and examples for how the SAFO framework can be applied within different CBL settings.

4.1. Descriptive Findings

We are careful not to interpret our findings conclusively, as we only have descriptive statistics of the students’ responses. In future studies, we will apply the framework to conduct inferential statistical tests. We highlight a few interesting findings from the current study.

First, IST scores ranged from 0 to 1 for both ‘adherence’ and ‘correctness’, and across all eight system elements. In other words, most study participants seemed to have a less-than-perfect level of IST across the board. With a lack of systems thinking frameworks for instructors to implement in first-year STEM education [3], and considering the study participants were first-semester students who had not encountered the SAFO framework previously, we can assume that most if not all of the study participants were not exposed to systems thinking in a formalized way prior to this study. This assumption may be confirmed based on analysis of Danish high school STEM curricula.

Second, all eight system elements received ‘adherence’ scores of 1, except for ‘benefit’, which received a score of 2. Lavi et al. [17] reported a similar result, where student teams scored higher for systems thinking on an element pertaining to the purpose (or benefit) of the technological system they were describing versus every other element of the system. It may have been easier for participants to intuit what ‘benefit’ meant, whereas other terms were either new to them or they were presented to them with meanings that were new to them.

Third, all eight systems elements received IST scores of 1, except for ‘output’ and ‘problem’, which received IST scores of 0. When looking at the ‘adherence’ and ‘correctness’ scores for both system elements, the difference in the IST scores stemmed from the ‘correctness’ scores, with a median of 0.5 for ‘output’ and 0 for ‘problem’. A potential reason for this was how examples for these elements were phrased in the adapted article: the problem being addressed by the SoI was described as more of a lack or a need, and the outputs and output systems of the SoI were implicitly described. We may revise the adapted article for future studies so that these elements are more clearly included.

4.2. Summative versus Formative Assessment of Introductory Systems Thinking

While the instrument allows for summative assessment of STEM students’ IST, providing insights into students’ current perspective and understanding of a system, which could be particularly useful in STEM education research, a summative assessment might not be the most relevant or valuable application of the SAFO framework to teaching practice, particularly in first-year education. Rather, it will allow instructors to provide students with a formative assessment in the form of targeted feedback concerning students’ understanding (and potential misconceptions) of the system aspects (architecture, function, and outcome) and of the IST criteria (adherence and correctness). The rubric and scoring guidelines could also be provided to students to support peer feedback and reflection.

4.3. Describing versus Prescribing a System

In this study’s article assignment, the SAFO framework was applied as a means for students to describe an existing system, and students could extract all the necessary information from the adapted article. However, the framework could potentially also be used to suggest improvements to the design of a system, such as increasing its benefits or reducing its detriments, e.g., by replacing parts. Furthermore, the SAFO framework could also be applied for prescribing an entirely new system, starting with the desired outcome, and then prescribing the system’s function and finally, its architecture. This would often be relevant in PBL settings, where the learning’s point of departure is a real-world problem, to which students are tasked with proposing, designing, and evaluating a solution.

4.4. Variation and Progression in Complexity and Learner Autonomy

While the SAFO framework can be applied in many learning settings, both in and outside of CBL, in the following, we will focus on potential implementations in active learning settings based on the former CBL typology (Table 3). While the framework is particularly useful in learning settings with high learner autonomy, such as PBL, we argue that a gradual increase in complexity is suitable, ensuring a minimum level of IST from which progression in systems thinking can be scaffolded through variations in the application of the SAFO framework.

Table 11. Potential applications of the system Architecture-Function-Outcome framework in case-based learning settings.

Case-Based Learning Type Student Autonomy Level	Learning Objectives	Instruction and Assessment
Case presented in class(lecture-based and/or directed) Very low or low autonomy	1. Understand the need for systems perspectives in engineering; 2. Acquire basic understanding of systems thinking; 3. Understand how the SAFO framework can be applied to describe a system of interest (SoI).	Describe systems thinking and the SAFO framework with examples; Provide SoI as a case; Students apply the SAFO framework to case and map SoI and its boundaries; Facilitate classroom discussion about potential improvements for SoI (e.g., to increase benefit or reduce detriment); Students take knowledge quiz.
Case presented as workshop/group work (interrupted and/or jigsaw) Medium or high autonomy	Same learning objectives as above and 4. Identifying an SoI; 5. Understanding perspective-taking in systems thinking; 6. Applying a systems thinking perspective in technology assessments and design processes.	Describe the SAFO framework and partial details of the case (e.g., adapted article) to groups of students; Students define and describe the SoI themselves using the SAFO framework; ‘Obstructions’ can be added, e.g., a particular detail of the case that needs to be integrated or new groups are formed to integrate identified SoI as part of new SoIs; Students propose solutions to improve the new system (e.g., increase benefit or reduce detriment); Solutions are evaluated through formative and peer feedback.
PBL project, with point of departure often from real-world problems, organized around team-based project work [29,30] Very high autonomy	Same learning objectives as above and 7. Applying systems thinking to authentic problems; 8. Applying a systems thinking perspective in negotiation and decision-making processes. Intended learning outcomes address both problem and process perspectives [36]	Student groups define their own case from an overall theme. There is no lecture, or the lecture only pertains to the SAFO framework. Each group applies the SAFO framework (or another framework) to support them in describing the problem and possible solutions from a systems perspective. Projects are assessed formatively (e.g., through a supervisor or peer group) and summatively (e.g., in a group exam).
System project or megaproject, with point of departure often from societal grand challenges, on connected problems that cannot be solved by a single team [37,38] Very high autonomy to self-directed	Same learning objectives as above and 9. Applying systems thinking to analyze the inter-relatedness of problems and systems; 10. Applying a systems thinking perspective to communicate across disciplinary boundaries; Intended learning outcomes address interdisciplinary and inter-team problem and process perspectives [39].	Interdisciplinary student groups or inter-teams define their own case from theme or challenge. Each group works on different and inter-related aspects of SoIs and use the SAFO framework to communicate with one another. Teams are supported through guided activities, workshops, and interdisciplinary teams of supervisors. Projects are assessed formatively and authentically (e.g., in relation to and by other boundary system projects, and with actual stakeholders).

below shows examples of such implementations with varying degrees of complexity and ordered by level of learner autonomy. Where possible, we provided examples from real-world learning experiences at Aalborg University in Denmark.

5. Conclusions and Future Work

In this section, we discuss the benefits and challenges of using the SAFO framework in first-year STEM education through CBL, outline the limitations of our study, and propose future research directions, and describe the methodological and practical contributions of our study.

5.1. Fostering and Assessing Introductory Systems Thinking in Case-Based Learning

Systems thinking is increasingly being recognized as a crucial skill across STEM disciplines due to its critical role in understanding and addressing complex, multifaceted problems [11,12,13,14]. A previous version of the SAFO framework has been used to foster and assess systems thinking of STEM novices—introductory systems thinking [3]. In this work, we designed an assignment based on the SAFO framework and developed a detailed assessment rubric suitable for both teaching and researching students’ IST in a wide variety of case-based STEM learning settings.

Introducing systems thinking to first-year students presents challenges, primarily due to their limited disciplinary knowledge and lack of prior exposure to complex systems analysis. Utilizing CBL variations as mediums, educators can provide experiential learning that deeply ingrains systems thinking competencies and, pending further validation, potentially use the SAFO framework to assess the impact of these learning experiences on students’ systems thinking, or as a framework for peer feedback and assessment, or even self-evaluation. Pedagogical innovations such as the SAFO framework are crucial for effectively preparing future STEM professionals and assessing their ability to tackle real-world and interconnected complex challenges.

5.2. Contribution to Research and Practice

As a methodological contribution, we showed how an assessment rubric can be developed for a thinking skill such as systems thinking, by separating the assessment of students’ adherence to the instructions (based on the construct of the skill) from the assessment of the correctness of students’ responses. Practically, this separation allows not only for a more precise summative assessment, but more importantly, it allows instructors to provide students with a formative assessment in the form of targeted feedback concerning the students’ understanding of system aspects (architecture, function, and outcome) and of the IST criteria (adherence and correctness).

Along with the practical contribution to teaching and assessing IST in practice in first-year STEM education, the current study provides a first step toward using the SAFO framework for exploring and evaluating progression in students’ systems thinking.

5.3. Limitations and Future Directions

The current study focused on the design of an assessment rubric based on the SAFO framework, establishing the inter-rater reliability of this rubric in a directed CBL setting, and suggesting potential CBL applications for this framework across the learning autonomy spectrum (Table 11). Future studies could implement one or more of these applications and use the SAFO framework and assessment rubric to evaluate their efficacy for fostering students’ systems thinking in different active learning settings.

In this study, we did not analyze the validity of the SAFO-based assessment rubric or other forms of reliability aside from the inter-rater reliability. In future studies, we may analyze the rubric’s construct validity using confirmatory factor analysis or its criterion validity using self-reporting measures, like in the study in which the previous version of the SAFO framework was applied [3].

Furthermore, while STEM education can take the form of disciplinary education, systems thinking is inherently transdisciplinary, and thus the SAFO framework might assist, pending further validation, with the scaffolding of collaboration, communication, and understanding across disciplinary boundaries. However, the efficacy of the SAFO framework in improving student skills and performance was not tested in this study. To further research the potential of the SAFO framework in interdisciplinary collaboration, future studies could implement a learning design such as a system or megaproject (see Table 11) to explore and assess the extent to which SAFO framework-based assignments scaffold interdisciplinary competences such as collaborative problem design, disciplinary reflection, and meta-competence [38].

Finally, future work might include the exploration of digital technologies to further augment and elevate the scaffolding and assessment of students’ IST, e.g., by using generative artificial intelligence-based tools to perform automated analyses of student responses to the SAFO framework-based learning assignments and provide comprehensive and timely feedback to students during student-led work, or to assist educators in generating exemplary cases for teaching IST suitable for various CBL settings.