1. Introduction
Complex systems (CSs) refer to a type of system characterized by numerous interacting elements that collectively achieve a holistic outcome [
1,
2,
3,
4]. Many STEM practitioners and researchers deal with and study CSs due to the prevalence of CSs in our world and the fact that many imperative issues, such as climate change, pandemics, biodiversity loss, and energy consumption, emerge from this type of system [
2,
4]. Accordingly, developing learners’ CS knowledge and the ability to understand CSs has gained increasing attention in STEM education since the last century [
5,
6,
7]. For example, the
Benchmarks for the Science Literacy Field [
8] include understanding systems and applying a systemic perspective to explain natural phenomena and solve real-life problems as a major science learning goal. The current U.S. K-12 science education standards,
Next Generation Science Standards (NGSS), include systems and system models as one of the crosscutting concepts in science learning and stipulate that U.S. secondary students should be able to utilize mathematical and computational tools to understand CSs [
9]. At the postsecondary level, CS knowledge and the ability to make sense of CSs have been defined as core competencies for undergraduates in various STEM disciplines [
7,
10]. Similar educational emphases can be identified from the science education documents in other countries, such as Germany and The Netherlands [
11,
12].
Education researchers contend that instruction must be deliberately designed and implemented to develop and broaden students’ CS knowledge because such knowledge does not come naturally for students and often is not developed through daily experience [
5,
13,
14,
15,
16,
17,
18]. Unfortunately, educators face a range of obstacles to providing rigorous instruction on CS knowledge. First, the current U.S. science education standards have not clearly defined what a complex system is. The phrase “more complex systems” is used to vaguely refer to an array of mechanical, social, and biophysical systems. These systems may be classified into complicated systems and complex systems as they present some distinct features [
3,
19]. Second, the instruments to assess one’s CS knowledge are limited. Interviews and artifact analyses (e.g., analyses of concept maps or drawings) were primarily used before now to reveal people’s understanding of CSs [
1,
17,
20]. We only identified a few relevant studies using or developing survey instruments for CSs. Dori et al. [
21] distributed a survey to systems scientists to explore their worldviews and assumptions related to systems, and Dolansky et al. [
22] reported the development of a systems thinking scale to measure systems thinking for healthcare professionals. Currently, there are no survey instruments available for science educators at the secondary and postsecondary levels to quickly assess their students’ CS knowledge. Third, secondary in-service teachers’ understanding of CSs largely remains underreported as the relevant studies disproportionately focused on students [
6,
23]. The existing studies suggest that many teachers possessed incomplete CS knowledge, which appeared to lower their self-efficacy and impede their abilities to teach these topics in classrooms [
24,
25,
26,
27,
28]. Given that secondary science teachers play a crucial role in setting the foundation on which to formally develop students’ CS knowledge and ability to think of CSs, further studies on secondary science teachers’ CS knowledge are needed.
This article presents an exploratory study that aimed to develop and validate an instrument, the Complex Systems Knowledge Survey (CSKS), based on the existing systems theories and the literature, to examine the CS knowledge among high school teachers and undergraduates in the United States. This effort also aimed to answer the following research questions:
What are the characteristics of CS knowledge among high school teachers and undergraduates?
How do the views of complex systems vary between high school teachers and undergraduates? What kind of heterogeneity in views exists within the group of teachers and undergraduates?
To what extent can high school teachers and undergraduates discern between complex systems and complicated systems?
Which systems do high school teachers and undergraduates regard as examples of complex systems?
3. Students’ and Teachers’ Understanding of CSs
Decades of studies have suggested that people often do not develop accurate CS knowledge from daily experiences. For example, prior to formal instruction, students at all ages were often found to primarily focus on the individual components of a CS [
35], misconceived the simultaneous, continuous interactions within a CS [
36], failed to identify the feedback processes [
28], and failed to recognize the emergent, dynamic nature of system outcomes [
1]. Studies also showed that adults still possessed misconceptions similar to those of youths. People with limited CS knowledge tended to possess a reductionist perspective, assumed centralized control and linear relationships, had difficulties in recognizing feedback, were confused by emergent levels, and failed to consider causal processes in time [
37,
38,
39].
A vast number of studies have been conducted to investigate students’ views and understandings of CSs at the secondary level and higher education [
6,
23]. In addition to the studies mentioned above, Yoon et al. [
40] examined secondary students’ learning progression for CSs. They found that students could easily comprehend the concepts of levels and the interconnected nature of systems but had difficulties understanding the concepts of decentralization and the unpredictability of CSs. After analyzing the explanations for behaviors of sand dunes from secondary students, undergraduates, and graduate students, Barth-Cohen [
41] found that, regardless of academic levels, students often started with centralized thinking while combining centralized, transitional, and decentralized components in their explanations. Due to the built-in uncertainties and disorders, the CS outcomes cannot be precisely predicted. Unfortunately, students often have difficulties understanding this notion [
1]. In a study on building a biology concept inventory, Garvin-Doxas and Klymkowsky [
42] found that the majority of undergraduates failed to recognize that randomness exists in complex systems all the time and it can influence emergent system behaviors. In addition, Talanquer [
43] surveyed and interviewed undergraduate students enrolled in an introductory chemistry course and reported that the undergraduates commonly used an additive, rather than an emergent, framework to predict the sensory properties of chemicals.
The studies on teachers’ understanding of complex systems are limited compared to those on secondary and postsecondary students. Only a few studies have reported on the trends in secondary teachers’ understandings. Sweeney and Sterman [
28] interviewed 11 middle school teachers and found that only 32% of them could recognize feedback structures. Skaza et al. [
27] surveyed 81 high school teachers who taught system models in their courses. They found only 34–35% of the teachers could correctly identify the causal relationships in the tested systems. Goh [
25] investigated 90 high school science teachers’ CS knowledge and found teachers had the most difficulties in understanding the dynamic nature of CSs. Teachers’ CS knowledge influences their instruction. Studies have found that teachers with a better understanding of certain CS concepts are more likely to integrate those concepts into their instruction [
24,
25]. In summary, the existing literature justifies the need for more studies on teachers’ knowledge of CSs.
6. Results
6.1. Instrument Validation
The PLS analyses resulted in the retention of 20 Likert-scale items (
Appendix A). Ten Likert-scale items were removed due to low non-significant outer loadings and their vague language, which did not clearly reflect the essential aspects of the constructs. For example, we removed the item “it takes time for a complex system to exhibit a change” on the basis that it was vague, in particular, because the respondents had no reference for how to define the time duration in the statement. The outer weight and loading of the item were also low and insignificant. In contrast, 19 of the 20 retained items had a significant indicator weight or loading. We kept item 18 despite its low non-significant outer weight and outer loading because it captured the essential aspect of stochasticity. No collinearity issues were detected among these 20 items (
Table 3). The response data were provided in the
Supplementary Materials (Table S1).
The path coefficients and R
2 of a structural model suggest the extent to which one variable affects another variable and, therefore, inform us of the variable relationships of the structural model [
54]. When building the relationships among the five constructs, we chose the directionalities among the five constructs based on two considerations. The first is the complex system theoretical framework. For example, we directed the element construct to the other four constructs because system elements serve as the foundation of a system; we directed the other four constructs toward the emergence construct because emergence describes the system properties resulting from other aspects collectively. The second consideration was based on the relationships we aimed to investigate. In this study, we were interested in the effects of respondents’ knowledge of micro-interactions on their knowledge of decentralization and stochasticity. Therefore, we pointed the model arrows from the micro-interaction construct to decentralization and stochasticity.
The results based on 10,000 bootstrapped samples showed that all the path coefficients (P) among the five constructs were significant and greater than 0.1, but the R
2 values were low, ranging from 0.128 to 0.488. The element construct exhibited a large effect on micro-interaction (P = 0.541), a medium effect on decentralization (P = 0.302), and a low effect on stochasticity (P = 0.230) and emergence (P = 0.189). Micro-interaction exhibited a weak effect on both decentralization (P = 0.276) and stochasticity (P = 0.176). Although emergence was significantly affected by the other four constructs, only the micro-interaction construct exhibited a medium effect (P = 0.414), whereas element, decentralization, and stochasticity only displayed weak effects (
Figure 1). The internal reliability analysis of the 12 binary complex system identification items produced a result of 0.606, which is acceptable for an exploratory study like this one [
51]. However, we failed to establish the convergent validity because the path coefficients between the five constructs and the system identification variable were insignificant (
Figure 1) and the average variance extracted (AVE) was only 0.133.
Although convergent validity was not established between the exogenous variables (i.e., the five CS constructs) and the endogenous variable (i.e., the system identification ability), the significant indicator weights and loadings suggested that most of the survey items strongly contributed to these variables. Therefore, we proceeded to use these retained items to examine the high school teachers’ and undergraduates’ CS knowledge.
6.2. Characteristics and Differences of Teachers’ and Undergraduate Students’ Knowledge of CSs
As indicated by their total percentage scores, both teachers (M = 80.90%, SD = 9.14%) and undergraduates (M = 72.14%, SD = 12.24%) possessed a moderate level of CS knowledge. The teacher group’s scores were significantly higher than those of the undergraduate group, with a moderately high effect size (Cohen’s
ds = 0.784). Both teachers and undergraduates received high scores on element, micro-interaction, and emergence, while scoring low in decentralization and stochasticity. Teachers outperformed undergraduates across all five constructs. However, the effect sizes showed that the knowledge differences between the two groups were weak in element and decentralization, while they were moderate in the other three constructs (
Table 4).
Within the teacher group, no significant differences were identified among the teachers in different subject areas, although science teachers received slightly higher total scores. Within the undergraduate group, undergraduates with a STEM or STEM-related major received significantly higher scores on total, element, micro-interaction, and emergence than the undergraduates with a non-STEM major. The effect sizes showed that the knowledge differences between the STEM and non-STEM groups were moderate to low. No significant differences in decentralization and stochasticity scores were found between the two groups (
Table 5).
When comparing non-STEM and STEM undergraduates based on years, statistical differences were only identified in freshman and sophomore students (
Figure 2). One-way ANOVA suggested significant differences in scores for total, element, decentralization, and micro-interaction for students in different years. Post hoc comparison using Tukey’s HSD test indicated that (1) the freshmen students received a significantly higher total score (M = 74.36%, SD = 10.82%) than the sophomores (M = 69.50%, SD = 13.61%) and juniors (M = 69.35%, SD = 13.02%); (2) the freshmen students received a significantly higher element score (M = 88.06%, SD = 14.87%) than the juniors (M = 80.36%, SD = 23.55%); (3) the freshmen received a significantly higher micro-interaction score (M = 81.93%, SD = 18.37%) than the sophomores (M = 75.50%, SD = 21.02%); and (4) there were no significant differences among undergraduates’ scores for decentralization and stochasticity. The effect sizes of the significant differences ranged from low to medium (
Figure 2).
6.3. Teachers’ and Undergraduates’ Identification of CSs
Most teachers and undergraduates could discern between simple systems and non-simple systems, and 33% of undergraduates identified a pendulum as a complex system, as indicated by only 23% of teachers. However, most teachers and undergraduates did not distinguish between complicated systems and complex systems because, although most teachers (73–98%) and undergraduates (77~91%) could successfully identify the seven complex systems, a large portion of teachers (77–96%) and undergraduates (68~91%) also identified the four complicated systems as complex systems (
Figure 3).
When ranking the 12 systems, the teachers appeared to have a better understanding of complex systems than the undergraduates. This trend was illustrated by significantly fewer teachers regarding a pendulum as a complex system than undergraduates (
Χ2 (1, 518) = 22.370,
p < 0.001) and significantly more teachers regarding traffic on a street (
Χ2 (1, 599) = 10.332,
p < 0.001), the Earth’s weather system (
Χ2 (1, 617) = 7.394,
p = 0.007), and a classroom (
Χ2 (1, 633) = 16.716,
p < 0.001) as complex systems than undergraduates. Nonetheless, significantly more undergraduates regarded chemical reactions as a complex system than teachers (
Χ2 (1, 596)
= 5.336, p = 0.021) (
Figure 3).
6.4. Complex System Examples from Teachers and Undergraduates
A total of 405 undergraduates and 242 teachers provided examples of complex systems. We identified three patterns among these examples. First, the respondents significantly more often cited social systems, with 425 mentions (63.4%, n = 670), and natural systems, with 360 mentions (53.7%), than mechanical systems, with 16 mentions (2.4%). Second, only a small portion of respondents provided their CS examples from more than one system category. Up to 57.8% of respondents’ examples fell into one of the three system categories (i.e., natural, social, and mechanical), while 29.3% of respondents provided examples from two system categories, often the natural and social systems. Meanwhile, seven respondents, 1%, provided examples from all three categories. Third, both complicated and complex systems were mentioned as CS examples, often without differentiation, instantiated by the following responses, “Cells, ant hills, human economies”, “The human body, corporations, and a community”, and “I thought of a large group of people like a sports team. Also I thought of different parts in a car that all work together”.
Chi-squared tests suggested that teachers and undergraduates were equally likely to mention natural, social, or mechanical systems. However, more teachers mentioned examples from two system categories than undergraduates (Χ2 (1, 583) = 19.408, p < 0.001). Significantly more science/STEM teachers mentioned natural systems than teachers in other subject areas (Χ2 (3, 252) = 19.409, p < 0.001), and significantly fewer science/STEM teachers mentioned social systems (Χ2 (3, 252) = 22.979, p < 0.001). Only science/STEM teachers provided CS examples from mechanical systems, such as cars, trains, etc.
STEM undergraduates were also significantly more likely to cite a natural system as the CS example (STEM = 59.7%) than non-STEM undergraduates (non-STEM = 46.7%, Χ2 (1, 418) = 7.102, p = 0.008), but there was no significant difference among STEM and non-STEM undergraduates in mentioning social systems as CS examples (STEM = 60.7%, non-STEM = 61.8%). There was also no significant difference in mentioning natural or social systems as CS examples among undergraduates at different years.
7. Discussion and Implications
The present study was an exploratory effort that aimed to develop and validate a survey instrument for assessing the knowledge of complex systems (CSs) among high school teachers and undergraduates. Our current work generated a 20-item survey questionnaire that measures respondents’ CS knowledge through five constructs. Furthermore, this research unearthed the attributes of CS knowledge among high school teachers and undergraduate students and the knowledge disparities between and within these two groups. Below, we discuss the insights gained in this exploratory work and connect the findings to the existing literature.
Developing a CS knowledge survey is challenging because CSs must be defined through multiple aspects, and each of these aspects possesses multiple features. We addressed this challenge by including the essential aspects of CSs as the measurement constructs and designing formative measurement indicators for each construct. Combining the PLS analysis results with complexity science theory, we identified 20 items that could be used to measure high school teachers’ and undergraduates’ CS knowledge about system elements, decentralization, micro-interactions, stochasticity, and emergence. The low variance inflation factor (VIF) values suggest these items may measure respondents’ CS knowledge independently. Nineteen of twenty items displayed significant indicator weights, suggesting they possess sufficient measurement quality. Hair et al. [
51] pointed out that an indicator with low non-significant loading can still be retained if there is strong theoretical support for its inclusion. Therefore, in this study, we retained item 18, “The outcome of a complex system can be predicted from individuals’ characteristics”, despite its low non-significant outer loading because it directly asks about the unpredictability of CSs, a key feature of stochasticity [
30].
Our structural model’s path coefficients and R2 produced important insights into the respondents’ CS knowledge. First, there were significant interconnections among respondents’ considerations of the five constructs, with the magnitudes ranging from small to high. Second, the ways in which respondents considered system elements influenced their views of the other four constructs, with the strongest impact on the micro-interactions and the weakest on emergence. Third, the respondents’ considerations of emergence were mainly affected by their views of system micro-interactions. Fourth, decentralization and stochasticity appeared to play a limited role in affecting respondents’ current CS knowledge, with us finding that they were weakly influenced and exhibited a weak influence themselves. We believe these findings are important for CS education because they present the first model regarding high school teachers’ and undergraduates’ CS knowledge based on the five constructs. Future investigations could involve testing this model with more respondents, and additional data could be collected to explain the magnitudes of the interconnections among these constructs.
The second challenge we faced was the lack of an existing survey instrument for CS knowledge. The survey questions used by Dori et al. [
21] do not differentiate among different types of systems. The other relevant scale [
22] measures systems thinking rather than CS knowledge. Therefore, we used respondents’ ability to identify CSs as an alternative reflectively measured variable [
51]. The limitation of this design is that the respondents may not identify CSs based on the formative constructs. The low convergent validity, low non-significant path coefficients between the five constructs and the CS identification variable, and low R
2 value suggest this is likely to be the case, because none of the five constructs affect how they identify CSs. Therefore, the next step to further this study would be to investigate the criteria respondents utilized to identify CSs.
Although a formative measurement model was not fully established in this study, the construct validity of the retained 20 formative measurement items and 12 system identification items was ascertained based on the low collinearity, significant indicator weights and loadings, and acceptable internal reliability. Therefore, although we may not have made a direct connection between respondents’ CS knowledge and their ability to identify CSs, we can still use these items to measure respondents’ CS knowledge and their ability to identify CSs separately.
We identified two significant findings on the respondents’ CS knowledge. First, both teachers and undergraduates participating in this study demonstrated relatively high knowledge of system elements and low knowledge of decentralization and stochasticity, and these patterns remained the same within teacher and undergraduate groups. Within the teacher group, no significant differences for the five constructs were identified in teacher CS knowledge across four disciplinary areas (i.e., science, math, social studies, and language arts). Within the undergraduate group, although STEM majors outperformed the non-STEM majors on the constructs of element, micro-interaction, and emergence, they both received low scores on decentralization and stochasticity. In fact, all undergraduates received low scores on decentralization and stochasticity regardless of their majors and years. More detailed information can be found by looking into the individual measurement items of the two constructs. For decentralization, most teachers and undergraduates believe “organizers” or “leaders” are necessary for CSs (merely 12.0% of undergraduates and 18.7% of high school teachers disagreed with this statement). For stochasticity, only 19.9% of undergraduates and 24.6% of high school teachers recognized the unpredictability of CSs. The second major finding was that the undergraduates possessed moderate to low knowledge of emergence. This was particularly evident in the non-STEM majors. Looking into the individual items for emergence, we found that only 48.1% of the non-STEM majors and 57.4% of STEM majors agreed that a CS may maintain stability when individuals within it constantly interact with one another. These findings on undergraduates are consistent with the previous studies that showed undergraduates did not possess a sufficient understanding of decentralization, stochasticity, and emergence [
40,
41,
42,
55,
56]. In regard to teachers’ knowledge of CSs, our findings add new knowledge to the existing literature by revealing the knowledge of decentralization and stochasticity that teachers in our study demonstrated.
There were additional nuances in the undergraduates’ understandings of the five constructs. Within the undergraduate group, STEM majors outperformed the non-STEM majors on the constructs of element, micro-interaction, and emergence, while both undergraduate groups received low scores on decentralization and stochasticity. It is interesting to see that freshmen students outperformed sophomores and juniors on the constructs of element and micro-interaction. This finding might indicate that freshman students received more instruction about systems at the high school level than the instruction sophomores and juniors received in their coursework. The results suggest that more concentrated instruction on CSs within high school and college courses is needed. Further investigations are necessary to reveal the underlying reasons for the differences in undergraduate students’ understandings of the five CS constructs.
Another salient contribution of this study is to reveal a common challenge among both teachers and undergraduates in distinguishing between complicated and complex systems, because most of the teachers and undergraduates identified all non-simple system examples as complex systems. This was also reflected by the fact that many teachers and undergraduates cited complicated systems when they were asked to provide examples of complex systems. We suspect this could be related to the lack of differentiation of complicated and complex systems in current science education standards and curricula.
Finally, we noted that, although teachers and undergraduates provided CS examples from both natural systems and social systems, the respondents who achieved a higher knowledge score, i.e., the science teachers and STEM majors, more frequently provided CS examples from natural systems. Further studies could be conducted on whether there is a correlation between exposure to complex natural systems and respondents’ proficiency in CS knowledge.
The current study has four main implications for science education in the United States. First, our findings suggest that there is a need for clear and precise definitions of complex systems within current science education frameworks. In the current U.S. K-12 science standards [
9,
14], the phrase “more complex systems” encompasses both complicated systems and complex systems. Based on teachers’ responses in our survey, the absence of a clear definition for complex systems, particularly the absence of the notions of decentralization and stochasticity, impedes teachers’ ability to effectively differentiate between the two types of systems. This may also prevent them from tailoring their instructional strategies. For example, human body systems (e.g., the digestive system, circulatory system, etc.) and ecosystems have distinct features, and teachers need to help their students understand these systems with different methods and learning objectives. At the secondary level, the definitions from Jacobson [
1] and Mitchell [
2] could be considered as they capture the essential features of CSs.
Second, educators must assess learners’ knowledge of CSs in terms of multiple aspects to reveal their existing knowledge. This study developed an assessment tool for educators to gauge learners’ CS knowledge in five aspects. We suggest that this tool be used before teaching CSs in science classes so that educators may gain a holistic understanding of their students’ CS knowledge and use it to inform their teaching.
Third, our findings suggest a pressing need for the deliberate design and implementation of enhanced learning experiences, including college courses and teacher professional development programs, to deepen their comprehension of CSs, especially the notions of decentralization, stochasticity, and emergence and their relationships. The survey scores directly show the low knowledge of decentralization and stochasticity among the teachers and undergraduates, while the weak path coefficients in the inner model suggest that these respondents did not closely connect decentralization and stochasticity to emergent system outcomes. The inner model and survey scores also show that, even though respondents appeared to possess an acceptable understanding of system elements and micro-interactions, such knowledge does not necessarily lead to a good grasp of decentralization and stochasticity. Therefore, explicit learning about these concepts is needed.
Fourth, on the bright side, our results show that teachers and undergraduates have a certain level of CS knowledge, including (1) viewing all five constructs as related, (2) possessing knowledge of the elements and micro-interactions of CSs, and (3) recognizing CS examples across natural and social systems. Curriculum developers and professional development facilitators may leverage this existing knowledge as a foundation upon which to build and expand undergraduates’ and teachers’ CS knowledge. Furthermore, the integration of appropriate technologies can effectively support learners’ comprehension. For instance, agent-based computer models have been successfully used to aid students in comprehending the concepts of decentralization and emergence in CSs [
57].
This study has limitations. First, the responses to the current version of the survey were gathered online, potentially limiting our ability to address any uncertainties or questions that respondents may have had regarding survey items. To address this, we will conduct ongoing interviews with a portion of the respondents to ensure the clarity and accessibility of the survey items. Second, the system identification items are binary. However, individuals’ perceptions of whether a system qualifies as a CS may fall in a spectrum. To address this limitation, we will replace the binary option with a Likert scale for the system identification items in the next version. Third, the current survey does not establish convergent validity. In subsequent studies, we plan to design items that directly capture respondents’ complex systems knowledge as the endogenous variable to facilitate the examination of convergent validity. Fourth, we used non-probability methods to distribute the survey to teachers, so the teacher respondents might not be highly representative of the high school teacher population in the USA. For example, only 12% of the teachers were in social studies. Additionally, both the teacher and undergraduate groups disproportionately included respondents who identified as female and white. Our audience must consider these limitations of the participants when interpreting our results.
Complex systems have gained rapidly increasing attention in STEM education over the past two decades [
23] due to their importance and prevalence in our lives. Our study aimed to devise an instrument for gauging CS knowledge, which will serve as the foundation for developing further cognitive procedures, such as systems thinking. We acknowledge that this effort marks merely the initial phase of our work. Through publishing this article, we intend to initiate dialogue and refine perspectives. We extend an invitation to STEM educators and systems educators to utilize this instrument now and engage in collaborative thought to advance this field of inquiry.