Comparing Students’ Multicultural Awareness Development in Online vs. Face-to-Face Diversity Courses Before, During, and After COVID-19

Abstract

Creating a global workforce is a driving need for our time. Higher education institutions are prioritizing helping students develop an awareness of their own and other cultures, as they are expected to participate in a diverse workforce. This study examines a large data set of learning assessment data for undergraduate students from an agriculture course focused on diversity and social justice, using the Beliefs, Events and Values Inventory (BEVI) instrument to assess the multicultural awareness development of the students. The study compared the impact of the medium of instruction (face-to-face versus online) in helping students develop multicultural awareness. The results revealed that students did not differ much in multicultural awareness across the medium of instruction. Moreover, to understand the multicultural awareness of the students in the pre-COVID period, during COVID and in the post-COVID period, the pre-test BEVI scores of the students were compared using ANOVA. The goal of this analysis was to capture the change in worldview of the students from the pre-COVID period to during COVID and post-COVID periods. The results revealed that pre-test BEVI scores for the pre-COVID period showed a lower degree of multicultural awareness than students during the COVID and post-COVID times. Also, there was no significant difference in the pre-test BEVI scores for the students in the COVID and post-COVID periods. Overall, the study makes important contributions to higher education literature as it reveals that (1) medium has negligible impact on the multicultural awareness of the students, and that (2) student worldviews have shifted significantly since the height of the global pandemic.

1. Background

Globalization and advances in transportation and communications technologies have increased the ability of individuals, groups, and institutions to connect across long distances. These individual, group, and institutional exchanges are often facilitated through travel and/or telecommunications platforms both domestically within countries and internationally. As a result, there is potential for increased diversity of interactions within and between regions. Diversity can be defined as real or perceived differences among individuals or groups [1]. These differences may be readily discernable in a tangible manner or manifest as underlying characteristics. With the increasing interaction with people from different backgrounds, it has become imperative for higher education institutions to acknowledge this shift and help to create a multiculturally aware and competent workforce. In recent years, there has been a proliferation of diversity-related programs in higher education institutions. The overarching goals of these programs are to improve diversity, equity, inclusion, justice, and to transform and create an inclusive learning environment where everyone belongs [2]

These programs, however, achieve mixed results since they deal with subject matter that can be highly charged to some and completely ignored or avoided by others [3,4,5]. Due to preconceived notions and beliefs about diversity, equity, inclusion, and justice (DEIJ), and worldviews that were formed prior to engagement with these programs, participants may experience different levels of engagement with the materials, concepts, and stakeholders associated with the programs [6,7]. Perry et al. [5], for example, noted student resistance, questioning of integrity, and devaluation in diversity education courses led by instructors of color. Iseminger, Acheson-Clair, et al. [6], however, concluded that instructors must find ways to minimize backlash effects on students who are resistant to diversity and social justice concepts in courses by addressing hyper-partisanship present in the classroom. Strategies such as understanding students’ prior experience with diversity and providing opportunities for reflection have previously been found to be effective in facilitating positive change in diversity consciousness and awareness in multicultural courses [7,8,9,10]. Overall, diversity courses have been found to improve students’ self-confidence, understanding of different perspectives, critical and creative thinking as well as understanding of self and others [11,12]. However, with the recent change of events due to the pandemic, we have experienced a major paradigm shift in how we think about enacting change to address systemic inequities, how to educate others about social justice issues, and how to facilitate the development of diversity awareness and consciousness [13,14]. Since this shift has impacted the lives of students, it is crucial to develop a theoretically grounded curriculum to help students reflect on the pressing social challenges and develop strategies to mitigate them.

The recent pandemic has also in many cases impacted the modalities of teaching, prompting instructors to consider strategies to engage students in online and face-to-face (F2F) instruction. COVID-19 significantly accelerated the shift towards distance learning [12]. This shift occurred alongside a broader societal and educational focus on diversity, equity, and inclusion, particularly within the United States higher education context. Such changes have spurred debates about the effectiveness of online learning compared to traditional face-to-face (F2F) instruction, especially in terms of fostering multicultural awareness. There are arguments supporting both sides; some advocate for the advantages of online formats in teaching multicultural awareness, while others prefer the F2F approach. For example, Singh et. al., ref. [13] argued that face-to-face settings are superior to teaching complex concepts, that the role of instructors is crucial in teaching such topics, and that the overall in-person learning environment plays an important role that fosters deeper engagement and more impactful outcomes. Despite the differing views, the pandemic forced educators to navigate the challenges of modifying their course content and teaching strategies for online instruction. In the beginning of the year 2020, nearly all colleges and universities had to deliver teaching and learning services using virtual platforms [14]. These virtual courses were either synchronous or asynchronous online, or a mixture of both forms [13]. The synchronous classes occurred in real time, with instructors teaching and students attending class simultaneously in different settings. These classes are similar to F2F classes in that class participation and discussions occur in real time, with fixed schedules. Asynchronous instructions, however, occurred on a staggered schedule, with students interacting with course materials and each other on their own schedule in different settings. As social distancing restrictions were relaxed, many courses were delivered as a mix of F2F and online delivery. According to Fabriz et al. [15], the asynchronous and synchronous teaching methods utilized in virtual setting during COVID-19 differed with respect to their potential to support students’ basic psychological needs and student social interaction, when considered from the students’ perspectives. Students studying in synchronous settings reported more peer-centered activities compared to those studying in mostly asynchronous settings, in addition to greater support for their basic psychological needs and greater overall satisfaction with the online environment. This disruption of traditional F2F models of delivering multicultural awareness curricula broke down resistance to online/virtual teaching and learning of social justice and diversity curricula. The questions were no longer should we do it or can we do it, but how do we facilitate intercultural learning effectively in an online/virtual environment?

Therefore, the goal of this study was twofold: (i) to explore the impact of the teaching modalities on the intercultural learning gains of students and (ii) to determine the change in worldview of students during the pre-COVID, COVID, and post-COVID periods. The study used the BEVI assessment to assess the intercultural learning gains of students enrolled in a multicultural awareness course offered in both online and face-to-face (F2F) formats. Further, the pre-test BEVI scores were used to compare the beliefs, values, and worldviews of the students who completed the course before the pandemic, during the pandemic, and after the pandemic. The goal was to compare the changes in beliefs, values, and worldviews for the three time periods. The research questions that we plan to answer through this study are:

RQ1: To what extent do learning outcomes in a social justice course vary across learning platforms?

RQ2: How do pre-test BEVI results, when compared before, during, and after the pandemic, reflect shifts in beliefs and values about oneself, others, and the world at large?

2. Materials and Methods

This study took place at a large, public, research intensive, Midwestern university. The research was conducted in a course listed as AGR 20100 Communicating Across Cultures (Course number, course name blinded). The course is open to all university students but is primarily populated by students from the College of Agriculture. The course seeks to satisfy the core curriculum requirement for multicultural awareness; to graduate, undergraduate students in the College of Agriculture are required to complete a 16-week-long course that requires students to meet for three hours each week.

2.1. Intervention

AGR course 20100 is a semester-long course that provides an opportunity for students to understand their place and others in the multicultural, multiethnic, multinational country of the United States. The course is designed as a basic overview of the variety of differences among human beings, a sampling that includes (but is not limited to) race, ethnicity, gender identity, age, ability, and social class. Issues of poverty, language, power, and oppression are also examined in relationship to these areas of emphasis [16] and the course addresses socialized prejudices, biases, and associated behaviors. The course is offered F2F and asynchronously online. The pedagogy includes invited expert lecturers, Socratic questioning, individual and group activities, community-based service engagement (Cultural Immersion Project—CIP), web-based curriculum exercises, and demonstrations, as well as YouTube videos, case studies, current events, and virtual reality simulations, among the authentic materials of culture that served as the basis of discussion and critical reflection.

Table 1. Course differences before and during COVID.

Face-to-Face		Online
Pre- and Post-COVID Period Fall 2017–Fall 2019; Spring 2022-Spring 2023	COVID-Period Spring 2020–Fall 2021 NOTE: Categorized as online medium for the purposes of analysis	Pre- and Post-COVID Period Fall 2017–Fall 2019; Spring 2022-Spring 2023	COVID Period Spring 2020–Fall 2021
Two weekly 50-min lectures delivered in person	Two weekly 50-min lectures live online	Two weekly pre-recorded video lectures.	Two weekly 50-min video lectures recorded from F2F class
Weekly, two-hour discussion-based labs/recitations facilitated in person	Weekly, two-hour discussion-based labs/recitations live online	Weekly discussion boards (initial post and reply to two classmates)	Weekly discussion boards (initial post and reply to two classmates)
Weekly quiz in lab section	Weekly quiz online	Weekly quiz online	Weekly quiz online
Midterm and final exams online (multiple choice)	No midterm or final exam	Midterm and final exams online (multiple choice)	No midterm or final exam
Five written reflections (three post-COVID)	Five written reflections	Five written reflections (3 post-COVID)	Five written reflections
Community-based immersion project activities	Remote community-based immersion project activities	Community-based immersion project activities	Remote community-based immersion project activities

shows a comparison of the online and F2F versions of the course pre-/post-COVID and during the COVID period.

As shown in Table 1, the primary changes to the course structure and facilitation before, during, and after the COVID period for both the F2F and online versions of the course were: during COVID, (a) F2F lectures and labs moved to fully virtual, (b) multiple choice midterm and final exams were removed for both classes, (c) the semester long community-based engagement project (CIP) activities were done remotely.

The CIP is a major component of AGR (Course number blinded). Students are assigned to small groups of 4–5 students at the beginning of the semester, and they work together to select a topic that addresses a particular social inequity topic covered in the course. The goal is to provide an experiential learning opportunity where students work with community organizations and/or campus partners and offices whose primary aims are to address specific inequities or injustices experienced by marginalized communities. Students gain first-hand experience and knowledge about the challenges experienced by members of the marginalized community and work to develop actionable recommendations to address the challenges alongside their community partners. The shift in the community-based activities being in a remote context meant that students in many cases could not engage directly with members of their community of focus or perform certain tasks that they would normally do such as serve in a homeless shelter, elderly care facility or food pantry. Depending on their topic, students engaged with their respective community partner organizations and by extension served members in their community of focus by conducting virtual interviews, attending virtual events, running social media campaigns or using flyers to raise awareness about the specific issues that members in their community of focus were experiencing (e.g., homelessness, food insecurity, gender pay gap, racial injustice, workplace discrimination by gender, sexual orientation, religion).

2.2. BEVI Instrument

Based on several decades of research including multiple factor analyses and other validation processes, the Beliefs, Events, and Values Inventory (BEVI) is an online multi-method (demographic, quantitative, and qualitative) assessment that holistically measures worldview [17]. The instrument’s underlying conceptual framework is the Equilintegration (EI) Theory, which aims to “explain the processes by which beliefs, values, and ‘worldviews’ are acquired and maintained, why their alteration is typically resisted, and how and under what circumstances their modification occurs” (p. 1075). EI Theory coalesces several principles from decades of developmental psychology scholarship, namely (1) that values and beliefs are not just are individual structures of the self but mediating processes shaping interactions and experiences at interpersonal as well as societal levels; (2) that values and beliefs are not necessarily grounded in logic or rationality, and may not even be recognized consciously by those who hold them; (3) that beliefs and values develop slowly over time as a result of people’s personal history, culture, and experiences; and (4) that because these development processes are highly complex and deeply embedded in people’s sense of self, the resulting beliefs and values are often difficult to change. The BEVI is a useful tool in educational settings because it can be leveraged both for formative assessment, where feedback is returned to learners to support the learning process, and for summative assessment, where data sets from before (time 1 or T1) and after (time 2 or T2) a learning experience can be compared for indications of transformative learning.

The BEVI is a psychometric instrument, meaning that responses to its 185 four-point Likert-scale items result in scores on 17 normed scales. The norming group for BEVI 3.0 is a global collection of more than 25,000 participants who are diverse in terms of age, gender, education, language, socioeconomic status, and many other aspects of identity. BEVI scores range between 1 to 100, and changes of at least five points from pre-post reflect some real-world meaningful shifts in beliefs and values. As illustrated in

Table 2. BEVI domains and scales of interest marked with directionality of desired change.

Domain	Dimension/Scale	Description	Desired Change
Core Needs	Needs Fulfillment	Open to experiences, needs, and feelings; deep care/sensitivity for self, others, and the larger world	Increased Scores
	Identity Diffusion (ID)	Indicates painful crisis of identity; fatalistic regarding negatives of marital/family life; feels “bad” about self and prospects	Ethnic Majority students—Increase Score preferred Ethnic Minoritized Students—Decrease Score preferred An increase in ID scores may be interpreted as good, especially in the short term, and particularly for students whose pre-test scores are low (often including those who represent an ethnic majority) as an increase in ID could be indicative of a state of questioning or reflection about their sense of self and their place in the world that could support their learning. In contrast, a decrease in ID may be seen as good for others, such as students who start out very high in ID (and are perhaps in an identity “crisis”) or those who belong to a marginalized group, as this decrease could reflect increased cultural self-awareness and self-acceptance, and a lessening of confusion or anxiety surrounding identity.
Tolerance of Disequilibrium	Basic Openness	Open and honest about the experience of basic thoughts, feelings, and needs	Increased Scores
Critical Thinking	Basic Determinism	Prefers simple explanations for differences/behavior; believes people don’t change/strong will survive	Decreased Scores
	Socioemotional Convergence	Open, aware of self/others, larger world; thoughtful, pragmatic, determined; sees world in shades of gray, such as the need for self-reliance while caring for vulnerable others	Increased Scores
Self Access	Meaning Quest	Searching for meaning; seeks balance in life; resilient/persistent; highly feeling; concerned for less fortunate	Increased Scores
	Self-Awareness	Introspective; accepts complexity of self; cares for human experience/condition; tolerates difficult thoughts/feelings	Increased Scores
	Gender Traditionalism	Prefers traditional/simple views of gender and gender roles	Decreased Scores
	Sociocultural Openness	Progressive/open regarding actions, policies, and practices in culture, economics, education, environment, gender/global relations, politics	Increased Scores
Global Access	Global Resonance	Invested in learning about/encountering different individuals, groups, languages, cultures; seeks global engagement	Increased Scores

, these 17 scales are organized conceptually into seven domains. It is important to note that high scores on these normed scales do not equate to good, ideal, desirable, or advanced. As a neutral instrument, the BEVI simply measures what is; the higher the score, the more of whatever is measured by that scale is present in the individual (or group, if it is a mean score). For example, Basic Determinism is a critical thinking scale that measures binary thinking and simplistic causal attributions. The higher the score, the more black-and-white a respondent’s thinking is in comparison to the norming group. Because the instrument is holistic, those interpreting BEVI data are encouraged not to isolate a single scale for sole analysis. However, because the data are so rich, often several scales of particular interest because of their relevance to learning outcomes become the focus of analysis.

2.3. Participants

Students enrolled in AGR 20100 Communicating Across Cultures from Fall 2017 to Spring 2023 comprised the participants in this study. This overall sample is divided into sections of the course taught F2F on campus in the pre-COVID period and those taught asynchronously via an online learning management system. During COVID, all versions of the course were delivered virtually through the same platform, including the same asynchronous version previously offered and a synchronous version of the earlier face-to-face course. The BEVI instrument administration system also automatically conducts six validity checks, including consistency and congruency of response patterns, missing items, and completion speed. The total number of participants for this study was N = 1807 students. Detailed numbers of participants by each time period, course medium, gender, and ethnicity are presented in

Table 3. Demographics.

Time Period	Instructional Medium	Gender				Ethnicity
		Total	Female	Male	Self-Identified Gender	White	Non-White
Pre-COVID	F2F	464	284	180	0	395	69
	Online	154	94	60	0	133	24
COVID	Online	664	412	246	6	545	119
Post-COVID	F2F	386	237	145	4	320	66
	Online	139	88	49	2	109	30

. The majority of participants were first- or second-year students at the time of data collection, and nearly all were students of Agriculture, as the course fulfills a Multicultural Awareness requirement within the college.

2.4. Data Collection and Analysis

This research was conducted at a large public research-focused university in the Midwestern United States. Both at the beginning (T1) and end of each semester (T2), the BEVI was given to all enrolled students in AGR 20100 as a required assignment graded only for completion. All students were provided customized secure login credentials to the BEVI dashboard, where they entered their responses and received their individual narrative reports of BEVI results. Group means were presented to students by a trained BEVI administrator after each data collection point in a reflective debriefing, and scale-level individual data were downloaded from the BEVI site by the authors for matched-pair statistical analysis. All analysis procedures were conducted at the end of the semester following approval from the human subjects research Institutional Review Board (IRB).

Based on the overall goal, content, and outcomes of the course we identified 10 of the 17 BEVI subscales to focus our analysis (see Table 2). The overall goal of the course is to provide students with opportunities to critically reflect on institutions and processes that contribute to the development of their values, beliefs, and worldview, as well as develop awareness and understanding of others with the goal of engaging in effective and appropriate interactions in a multicultural society. We used the Statistical Package for Social Sciences (SPSS) 28 to conduct different statistical analyses. First, we used descriptive analysis to examine aggregated pre–post results for the BEVI subscales across instructional mediums. The goal of performing the descriptive statistics was to identify the mean and standard deviation of the various BEVI constructs. The descriptive statistics helps to summarize and organize the data as it helps to identify the patterns and trends in the data set. Second, t-tests were used to examine whether there were statistically significant changes pre–post within and across the different mediums of instruction for subscales. A t-test was chosen to compare the online and face-to-face instructional mediums because it is specifically designed to assess differences between the means of two groups. Since our comparison involves two distinct conditions (online versus face-to-face instruction), the t-test is the appropriate statistical method. It allowed us to determine whether the differences observed in student performance between these two instructional approaches are statistically significant, and whether those differences are likely to be due to the instructional medium or simply occurred by chance. Third, one-way ANOVA was used to compare the pre-test scores of the students who completed the course during the pre-COVID, COVID, and post-COVID period. ANOVA is ideal when comparing the means of more than two groups. Since we are examining the pre-test scores across three distinct time periods, ANOVA is the appropriate statistical method to test for significant differences between them.

3. Results

3.1. Face-to-Face Versus Online Baselines

We used an independent t-test to assess the pre-test BEVI scores for the face-to-face and online mediums, see

Table 4. Pre-test score comparison for F2F and online medium.

Domains	Scale	Mean F2F	SD	Mean Online	SD	Diff (Online-F2F)	t	df	p
CORE NEEDS	Needs Fulfillment	47.66	29.07	49.52	29.97	1.85	1.33	1793	0.18
	Identity Diffusion	46.43	28.62	48.48	28.93	2.05	1.51	1786	0.13
TOLERANCE OF DISEQUI.	Basic Openness	55.10	27.80	57.28	27.70	2.18	1.66	1781	0.10
CRITICAL THINKING	Basic Determinism	46.98	29.21	42.87	28.06	−4.11	3.03	1764	0.00
	Socioemotional Convergence	48.68	28.73	50.08	28.76	1.40	1.03	1782	0.30
SELF ACCESS	Self-Awareness	51.96	28.43	52.94	29.01	0.97	0.71	1789	0.47
	Meaning Quest	46.43	28.07	47.68	29.30	1.25	0.92	1796	0.35
OTHER ACCESS	Gender Traditionalism	52.91	29.62	49.85	29.56	−3.06	2.19	1781	0.03
	Sociocultural Openness	43.57	30.85	46.78	32.07	3.21	2.16	1795	0.03
GLOBAL ACCESS	Global Resonance	39.34	30.05	40.53	30.80	1.19	0.82	1791	0.41

. The results of the analysis showed that students in the online medium had statistically significantly lower scores in Basic Determinism and Gender Traditionalism in comparison to the F2F students. Students in the online medium showed higher scores for all other scale criteria compared to F2F students, but only the higher score for Sociocultural Openness was statistically significant. All differences, even the three statistically significant findings, were small—that is, less than five points on the 100 point scales.

3.2. Face-to-Face Pre- and Post-Test Changes

A paired t-test was conducted to determine if there was any significant difference in the pre- and post-test scores for the F2F medium, see

Table 5. Pre-post score comparison of the F2F Medium.

Domains	Scale	F2F Pre (Mean)	StDev	F2F Post (Mean)	StDev	Diff Post-Pre	t	p
Core Needs	Needs Fulfillment	47.66	29.07	46.49	29.49	−1.17	0.89	0.37
	Identity Diffusion	46.43	28.62	48.79	28.64	2.36	1.83	0.07
Tolerance of Disequi.	Basic Openness	55.10	27.80	55.82	27.16	0.72	0.61	0.54
Critical Thinking	Basic Determinism	47.20	25.97	45.31	25.83	−1.90	1.67	0.10
	Socioemotional Convergence	46.98	29.21	49.65	30.50	2.67	2.05	0.04
Self Access	Self-Awareness	48.68	28.73	46.58	29.05	−2.10	1.65	0.10
	Meaning Quest	51.96	28.43	50.16	29.12	−1.81	1.43	0.15
Other Access	Gender Traditionalism	60.14	31.14	59.49	30.90	−0.65	0.48	0.63
	Sociocultural Openness	52.91	29.62	53.13	30.74	0.22	0.17	0.86
Global Access	Global Resonance	39.34	30.05	38.47	30.28	−0.87	0.65	0.52

. The analysis revealed that students showed a decrease in Needs Fulfillment, Basic Determinism, Self-Awareness, Meaning Quest, Gender Traditionalism, and Global Resonance as they moved through the semester. It is also important to note that none of these decreases in scores was statistically significant, whereas a statistically significant increase of 2.67 points was observed for Socioemotional Convergence, which suggests that the face-to-face medium had a measurable and meaningful impact on students’ ability to connect emotionally with others or align their emotions and social experiences. Also, a mean increase of 2.36 points in Identity Diffusion approached but did not reach statistical significance with p = 0.07.

3.3. Online Pre- and Post-Test Changes

The paired sample t-test was also used to understand the difference between the pre-test and post-test scores for the online medium, see

Table 6. Pre–post score comparison of online medium.

Domains	Scale	Online Pre	StDev	Online Post	StDev	Diff Post-Pre	t	p
Core Needs	Needs Fulfillment	49.52	29.97	49.75	29.59	0.23	−0.18	0.85
	Identity Diffusion	48.48	28.93	51.14	29.13	2.66	−2.14	0.03
Tolerance of Disequi.	Basic Openness	57.28	27.70	58.17	26.67	0.89	−0.77	0.44
Critical Thinking	Basic Determinism	42.87	28.06	45.42	29.27	2.55	−2.13	0.03
	Socioemotional Convergence	50.08	28.76	49.97	29.02	−0.11	0.09	0.93
Self Access	Self-Awareness	52.94	29.01	52.53	28.32	−0.40	0.32	0.75
	Meaning Quest	47.68	29.30	47.63	28.45	−0.05	0.04	0.97
Other Access	Gender Traditionalism	49.85	29.56	47.88	30.33	−1.97	1.55	0.12
	Sociocultural Openness	46.78	32.07	48.03	32.13	1.25	−0.92	0.36
Global Access	Global Resonance	40.53	30.80	41.91	31.58	1.38	−1.04	0.30

. Students showed a marginal increase on the majority of scales. However, a decrease in scores was observed for Socioemotional Convergence, Self-Awareness, Meaning Quest, and Gender Traditionalism but the decrease was not statistically significant. It is important to note that, of these changes, only the increases of 2.66 points in Identity Diffusion and 2.55 points in Basic Determinism were statistically significant. The statistically significant increase in Identity Diffusion may indicate that students became more aware of challenges in forming a stable self-identity, reflecting growth in their understanding of this concept. For Basic Determinism, the rise suggests a shift in students’ thinking toward more deterministic views, meaning they may have developed a stronger belief in predetermined or inevitable outcomes. These statistically significant changes are important because they indicate areas where the course had a measurable and meaningful impact on student development.

3.4. Comparison of Change Scores for F2F vs. Online Groups

The growth of the students from pre- to post-test for F2F and online mediums were compared. The difference between (a) F2F pre- and F2F post- and (b) Online pre- and Online post- were calculated and compared using an independent t-test, see

Table 7. Change score (post–pre) comparison of F2F vs online medium.

Domains	Scale	F2F Post-Pre	Online Post-Pre	t	df	p
Core Needs	Needs Fulfillment	−1.2	0.2	0.77	1792	0.44
	Identity Diffusion	2.4	2.7	0.17	1788	0.87
Tolerance of Disequilibrium	Basic Openness	0.7	0.9	0.1	1792	0.92
Critical Thinking	Basic Determinism	2.7	2.5	−0.07	1769	0.94
	Socioemotional Convergence	−2.1	−0.1	1.12	1786	0.26
Self Access	Self-Awareness	−1.8	−0.4	0.79	1793	0.43
	Meaning Quest	−1.2	−0.1	0.68	1795	0.49
Other Access	Gender Traditionalism	0.2	−2	−1.2	1788	0.23
	Sociocultural Openness	−0.1	1.2	0.71	1793	0.48
Global Access	Global Resonance	−0.9	1.4	1.19	1794	0.24

. The results of the analysis revealed that there was no significant difference between the BEVI scores for the two groups. This finding suggests that the mode of instruction—whether face-to-face or online—did not affect students’ growth or progress in these domains. The students in both environments experienced similar levels of intercultural development, meaning that the effectiveness of the learning experience was comparable across both mediums. This implies that both F2F and online learning are equally capable of supporting students’ intercultural growth in the assessed areas.

3.5. Comparison of Time Periods: Pre-, During, and Post-COVID

One-way analysis of variance (ANOVA) was performed to compare the pre-test scores for the students who took the BEVI in three different time periods, see

Table 8. Pre-test comparison of pre-COVID, COVID, and post-COVID time periods.

Scale	Pre-COVID		COVID		Post-COVID
	Mean	SD	Mean	SD	Mean	SD	Sum of Sq	Mean SQ	df	F	p
Needs Fulfill.	43.44	28.37	49.81	30.00	53.29	29.47	28,968	14,484	2	16.87	0.00
Identity Diff.	43.33	26.87	48.76	29.30	50.87	29.78	17,751	8875	2	10.82	0.00
Basic Open.	50.63	27.41	58.51	27.56	60.01	27.43	30,330	15,165	2	20.10	0.00
Basic Determ.	50.02	28.47	42.22	28.02	41.96	28.94	25,483	12,742	2	15.75	0.00
Socio. Conv.	45.19	28.40	50.59	28.83	52.93	28.49	18,436	9218	2	11.28	0.00
Self-Aware.	46.89	28.10	54.44	28.98	56.58	28.19	30,680	15,340	2	18.95	0.00
Mean. Quest	42.72	27.45	49.20	29.63	49.57	28.49	17,956	8978	2	11.10	0.00
Gender Tradit.	54.80	28.10	49.79	29.81	49.05	30.77	11,738	5869	2	6.73	0.00
Socio. Open.	40.29	29.95	47.47	32.46	48.35	31.50	23,547	11,774	2	11.99	0.00
Global Reson.	36.98	29.90	40.99	30.86	42.21	30.34	8854	4427	2	4.80	0.00

. The results of the analysis demonstrated that the time-based model was significant. The ANOVA examined the effect of pre-COVID, COVID and post-COVID periods on the BEVI scores of the students. The descriptive statistics for each BEVI scale for each time period are presented in Table 8. The results of the ANOVA revealed that time period had a significant effect on the BEVI scores of the students.

Further, a post-hoc Tukey HSD test was conducted, and it was found that there was no significant difference in the BEVI scores of the students for the COVID and post-COVID periods, see

Table 9. Tukey HSD test results.

	Pre-COVID–COVID			Pre-COVID–Post-COVID			COVID–Post-COVID
Scale	Diff. in Mean	t	p	Diff. in Mean	t	p	Diff. in Mean	t	p
Needs Fulfillment	6.37	3.89	0.00	9.85	5.66	0.00	3.48	2.03	0.10
Identity Diffusion	5.43	3.39	0.00	7.54	4.43	0.00	2.11	1.26	0.42
Basic Openness	7.88	5.13	0.00	9.38	5.75	0.00	1.49	1.6	0.62
Basic Determinism	−7.8	−4.9	0.00	−8.6	−4.77	0.00	−0.26	−0.16	0.99
Socioemotional Convergence	5.40	3.38	0.00	7.74	4.56	0.00	2.34	1.4	0.34
Self-Awareness	7.55	4.74	0.00	9.69	5.74	0.00	2.14	1.29	0.40
Meaning Quest	6.48	4.05	0.00	6.84	4.04	0.00	0.37	0.22	0.97
Gender Traditionalism	−5.01	−3.03	0.01	−5.75	−3.28	0.00	−0.74	−0.43	0.90
Sociocultural Openness	7.18	4.1	0.00	8.07	4.34	0.00	0.88	0.48	0.88
Global Resonance	4.02	2.36	0.05	5.23	2.9	0.01	1.21	0.68	0.77

. Comparison between the pre-COVID and COVID periods showed a statistically significant change in all the dimensions (of BEVI scales) for the COVID period, with increases in all scales except Basic Determinism and Gender Traditionalism. A similar trend was observed for the BEVI scores for the pre-COVID and post-COVID period students, revealing the same pattern—all scales were higher post-COVID than pre-COVID except Basic Determinism and Gender Traditionalism, and all differences were statistically significant. The results for the Tukey HSD test are in the Table 9 below. Also, Figure 1 illustrates significant shifts in various psychological and social scales across the BEVI across three time periods: Pre-COVID–COVID, Pre-COVID–Post-COVID, and COVID–Post-COVID. Most scales show substantial positive changes from Pre-COVID to COVID and from Pre-COVID to Post-COVID, with smaller changes observed from COVID to Post-COVID. For instance, Needs Fulfillment, Identity Diffusion, Basic Openness, Socioemotional Convergence, Self-Awareness, Meaning Quest, Sociocultural Openness, and Global Resonance all exhibit notable increases, suggesting that individuals experienced heightened fulfillment, openness, awareness, and sociocultural engagement during and after the pandemic. In contrast, Basic Determinism and Gender Traditionalism show significant negative shifts, indicating reduced belief in determinism and traditional gender roles over these periods.

Also, Figure 1 illustrates significant shifts across various BEVI scales over three time periods: pre-COVID–COVID, pre-COVID–post-COVID, and COVID–post-COVID. Most scales show substantial positive changes from pre-COVID to COVID and from pre-COVID to post-COVID, with smaller changes observed from COVID to post-COVID. For instance, Needs Fulfillment, Identity Diffusion, Basic Openness, Socioemotional Convergence, Self-Awareness, Meaning Quest, Sociocultural Openness, and Global Resonance all exhibit notable increases, suggesting that students experienced heightened fulfillment, openness, awareness, and sociocultural engagement during and after the pandemic. In contrast, Basic Determinism and Gender Traditionalism show significant negative shifts, indicating reduced belief in determinism and traditional gender roles over these periods. Overall the results suggests that the pandemic had a lasting impact on various aspects of psychological and social behavior.

4. Discussion

This study was centered around undergraduate students enrolled in the college of Agriculture in a large public Midwestern university. The study aimed to understand the impact of modalities of teaching (online versus F2F) on the multicultural awareness development of the students and also analyzed the pre-test BEVI scores of the students who enrolled in the college of Agriculture during the pre-COVID, COVID, and post-COVID period. The analysis of the pre-test scores across the three time periods was aimed at understanding the impact of COVID-19 on students’ intercultural competence and worldview development. The study used inferential statistics such as t-test to compare the impact of teaching modalities on intercultural development of the student. Further, one-way ANOVA followed by Tukey’s HSD post hoc test was used to assess the impact of the pre-COVID period, COVID period, and post-COVID times on the worldview development of the students. Returning to our guiding research questions, in this section we discuss both the explanation of the findings and the pedagogical implications of each question.

RQ1: To what extent do learning outcomes in a social justice course vary across learning platforms?

The short answer to this question is that the learning outcomes do not seem to vary much across platforms. While the individual analysis of each medium’s pre-/post-test comparison yielded some evidence of learning in each case, these increases and decreases in scores are always small, reaching statistical significance at times but not crossing the five-point threshold that the BEVI uses as a minimum difference indicative of real world meaningfulness. When the changes between pre- and post-tests for the two mediums are compared, these small differences do not emerge as significant. The data in this study showed no scales with statistically significant differences between F2F and online delivery modes (see Table 7), whether before/after the pandemic when students had a choice between the two modes of instruction or during the pandemic when all students were forced into the online delivery mode by external circumstances. Despite assumptions that are prevalent about the challenges of teaching/learning DEIJ content online, the online context in this study had similar outcomes to the F2F delivery.

The pandemic also did not significantly change the outcomes of the course, which was perhaps surprising given how stressful some COVID period experiences (e.g., switching mid-semester from F2F to online delivery, challenges with access to sufficient technological devices and internet services for effective connectivity to online platforms, student preferences for F2F versus online learning) presumably were for students and instructors alike. These findings highlight the difficulties of trying to document dramatic changes in values and beliefs stemming from a single semester-long course. The small changes from pre- to post-test in this study in both mediums align with expectations based on recent scholarship, as other studies have also noted that little or no change is realistic in pre-/post-tests over short time spans [18]. The findings are also supported theoretically. BEVI results in particular tend to be highly reliable over time, as reflective of Equilintegration Theory [17], which posits that once formed, the worldview webs made up of value and belief systems are stable and resistant to alteration.

The study further revealed an expected backlash effect of the DEIJ course content that also aligns with both EI Theory [19] and previous scholarship [20]. When cognitive dissonance is created for students as a result of engaging with cultural differences and examining power dynamics in the course, some typically react in defense of their current belief structures, moving away from rather than toward intended learning outcomes (for example, becoming more black-and-white in their thinking instead of having more grey areas, as reflected in the Basic Determinism scale). This backlash effect can be a temporary reaction to stress that resolves over time, as demonstrated by Wandschneider et. al [18]. It is important to note that drawing conclusions based on whole group aggregate analysis can be dangerous, since sub-group variation can be hidden within those whole group means. We know from past research that it matters who learners are when they enter the course, as their identity and previous experiences shape their reaction to the materials and their interactions with their peers, thus resulting in varying movement on BEVI scales from pre- to post-test. Iseminger et al., [20] offer a clear example of why sub-group analysis is so important for gauging course effectiveness for students of different social identities.

Regardless of whether BEVI scores for the students changed from the pre- and post-test, the instrument proved valuable as a formative tool. It enabled students to reflect on their worldviews, set learning goals, and identify changes in their thinking as the course progressed. Other scholars have noted the usefulness of instruments such as the BEVI for these purposes as a formative assessment in DEIJ or intercultural courses [21]. This has been the instructors’ qualitative experience in the AGR 20100 course as well—that the BEVI group debriefing is meaningful to students as a formative assessment because it makes value systems (the instructors’ and institutions’ in addition to the students’) and demonstrates the significance of students’ perspectives in comparison to their peers.

In this study, the assumption that online social justice course delivery would not be impactful was refuted. The authors would argue that the same guiding principles of learning in F2F contexts still apply to online learning; that is, good teaching and course design need to happen in any modality. Shifting cognitive, affective, and behavioral practices among learners to reach outcomes such as critical self-reflection, awareness of oneself and others as cultural beings, understanding different worldviews, curiosity, and openness towards others with different perspectives requires intentional practices of mentorship. Online or F2F, the research suggests that in order to facilitate development in cultural competence it is important to guide participants with appropriate levels of challenge and support as they engage in self-reflection opportunities to discuss their ideas with others, whether that be online through discussion boards or face-to-face in small groups [22].

Since the COVID period, definitions of online instruction have shifted as everyone was forced to be online regardless of how the course was initially designed. While some argue that the online medium has negative effects on some learners due to learning preferences, others argue that it is not simply the medium but rather the pedagogical practices and strategies employed that yield positive learning outcomes. In the COVID/post-COVID period of teaching diversity and social justice courses, there is a need to adopt pedagogical approaches that allow for learning across different modalities. While traditionally the approach to teaching and learning about DEIJ centers around having conversations F2F, due to the many challenges and concerns that emerge when using online platforms, the pandemic has shown us that we need to exercise flexibility and find new ways to effectively and appropriately structure arguments about DEIJ not only through F2F mediums but online as well.

RQ2: How do pre-test BEVI results, when compared before, during, and after the pandemic, reflect shifts in beliefs and values about oneself, others, and the world at large?

Pre-COVID, the BEVI results were highly reliable for pre-tests across cohorts. Figure 2 shows how little change there was from year to year before the pandemic across the relevant BEVI scales.

The overall pattern of BEVI scores stays consistent before the pandemic, as has been found in other large-scale institutional studies [18]. In this study, not a single scale shift from semester to semester from 2017 to 2019 or even cumulatively across all three years exceeds the benchmark for real-world meaningfulness of a five-point mean difference on the 100-point normed scales. Then, there was a sudden dramatic shift in response to global events and experiences during the COVID period, with statistically significant and real-world meaningful shifts of five to 10 points on every single scale (see Table 8). Furthermore, the change in values is holding over time instead of reverting back to the scores consistently seen pre-pandemic (see Table 9). Interestingly, all of these shifts are in the direction of intended learning outcomes for the AGR 20100 course, as outlined in Table 2. Thus, post-pandemic, learners are entering the course already closer to the learning outcomes than previous cohorts of students had been prior to the COVID period.

These sharp shifts are also all in alignment with Equilintegration Theory [17]. The BEVI Full Scale Score is a composite score drawn from 10 of the 17 BEVI scales, some of which contribute positively to its calculation and others negatively, for both theoretical and statistical reasons [19]. The BEVI Full Scale Score has been used as a proxy for Global Citizenship and other pro-social constructs [23], and generally reflects high-functioning human beings with strong capacities for engaging with self, others, and the larger world [18]. In remarkably consistent patterns, scales that positively contribute to the BEVI Full Scale Score are moving up (illustrated in Figure 3) and scales that negatively contribute to the BEVI Full Scale Score are moving down (see Figure 4). Thus, the data in these figures present a clear picture of a population moving towards optimal being-in-the-world, greater self-efficacy, and enhanced capacity for positive relationships over the course of several years, in response to the stimulus of the historically unique experiences of the COVID period.

Values and beliefs are fairly stable once constructed and often need traumatic events to substantially alter them, according to TL theory [24]. Important to understanding what is happening with this clear shift in worldviews over time in response to the pandemic is the explanation that Equilintegration Theory provides about how value systems are shaped by how people’s needs are met (or not) [17]. We must therefore take into account not just historical events of the time but how people reacted to those events in attempts to meet their physical, emotional, and social needs. For instance, the global pandemic brought undeniably dramatic changes in lifestyle, remote work/school, social connection/isolation, and health and relational stressors; to counter these negative impacts on our capacity to meet our own and others’ needs, though, a social shift towards mental health awareness, the promotion of self-care practices, and an emerging field of telepsychiatry materialized [25]. At the same time, increased media coverage of racialized violence led to greater recognition of social injustice and the need for DEI work among the majority population [26]. Books on anti-racism, e.g., Kendi [27], and white fragility, e.g., DiAngelo [28], became popular choices for book clubs and discussion groups, and anti-Asian violence sparked bystander training (e.g., RightToBe, 2023) as a common professional development option. Meanwhile, political polarization during the 2020 election and its aftermath (the events of January 6, etc.) brought US youth into greater political and identity awareness during this time [29].

Post-pandemic, students who enroll in the course come in with increased levels of multicultural awareness, starting closer to intended learning outcomes than previous cohorts. As a result, course design and pedagogical strategies need to be tailored to address their changing needs and preparedness for learning. This simultaneous challenge and prospect has prompted a need to operationalize student agency by providing opportunities for students to hold space and to define learning in their own language. Therefore, it has become even more vital to incorporate multiple modalities to collect evidence of student learning and validate pluralistic ways of gaining and demonstrating knowledge. For example, during the COVID period, the pedagogy of service-learning had to be restructured since students were not able to conduct in person site visits with their community partners. The modification consisted of creating virtual opportunities for students to engage in the multiple dimensions of their service-learning project. Since service-learning is a reciprocal process, the importance of addressing community partner identified needs became a high priority.

5. Conclusions

This study leveraged a large, combined sample of pre- and post-tests from multiple cohorts of learners in the same course, collected before, during, and after the pandemic period. This study design itself is a unique contribution. The big data approach contrasts with the more typical study comparing whole group aggregate pre-/post-test change means for a single cohort. However there are certain limitations to this study. First, at one glance (with aggregate means), there is not a lot of quantitative evidence of a value shift among students in response to the course experience. This finding is typical of a short learning experience and a bird’s eye view with whole group means. One useful approach to explore learning outcomes of a course like AGR 20100 would involve T3 or delayed post-test, since short-term backlashes to challenging learning experiences can resolve over time [18], and another option would have been to explore detailed sub-group analyses to find patterns within group variation [22]. However, both of these research design elements were beyond the scope of the current project. Secondly, the study would have benefited from an analysis of qualitative data that could potentially show developmental growth, perspective changes, and the impact of social identity on learning experiences. Some limited qualitative data are gathered by the BEVI instrument and there are learning artifacts from the course that could be examined, but such data could not be treated sufficiently in the current article.

Based on these contributions and limitations, we suggest the following as potential new directions for future research:

• Statistical regression with similar large data sets to determine the impact of social identities on learning outcomes—do different sub-groups perform better in different platforms?
• Analysis of rigorous qualitative data for phenomenological understanding of DEIJ course experiences, including the benefits and challenges of online learning for this content—as examples, online discussion board content could be compared with recorded face-to-face discussions in labs, or focus groups could discuss their learning experiences in these different platforms.
• Additional big data studies of the impact of the pandemic period on values and beliefs—in this data set the shift in values seems to be lasting rather than a temporary blip in reaction to pandemic circumstances with a later return to pre-COVID BEVI scores. As we move on from the pandemic period, does the change seem permanent, and if so, what implications do these shifts in beliefs and values hold for DEI learning and higher education in general?

Overall, based on the findings of the study we can conclude that the study provides an important model for rigorously assessing learning outcomes in a DEIJ course, as so many of these training experiences are assumed to be effective rather than demonstrated to be effective. Furthermore, the study fills a gap in the literature by offering evidence that DEI online learning outcomes are not all that different from face-to-face learning outcomes. Instead, it would seem that the quality of instructional design and delivery matters more than the medium of instruction. The discussion highlights that in the relatively unknown territory of delivering these courses completely online on a large scale during the pandemic, pedagogical implications needed to be taken into account for courses of this type to be as successful as possible. Finally, the study is significant to contemporary scholarship because its longitudinal data clearly demonstrate the impact of COVID period experiences (including not only the pandemic but the racialized and political events of the time) on the collective consciousness and value systems of the target population, university students in the United States.

Implications for Teaching and Learning

The implications for teaching and learning in undergraduate education are far-reaching, particularly in the context of Diversity, Equity, Inclusion, and Justice (DEIJ) courses. The study underscores that the quality of instructional design is crucial, suggesting that whether a course is delivered online or in person, the instructional methods, learning objectives, and student engagement strategies must be carefully crafted to ensure effective learning. This is particularly relevant for undergraduate education, where students are often in their formative years of developing critical thinking skills, cultural awareness, and ethical decision-making abilities.

In undergraduate education, where students are preparing for diverse professional environments, integrating current sociopolitical contexts into DEIJ courses is essential. Students need to engage with real-world issues, such as racial justice movements, gender equality, and other social justice matters, to make the learning experience relevant and impactful. By weaving these topics into the curriculum, educators can foster a deeper understanding and reflection, enabling students to critically examine their own beliefs and the world around them. For example, discussions about the Black Lives Matter movement, climate justice, or immigration policies can make DEIJ courses timely and resonate more deeply with students.

The study also highlights the importance of sustained learning, particularly through longitudinal assessments. In undergraduate education, this means moving beyond the traditional end-of-course evaluations to implement long-term assessments that track students’ growth over time. These could include follow-up surveys or reflective assignments, assessing whether students have integrated DEIJ principles into their personal and professional lives after the course has ended. This is especially crucial in undergraduate education, where the goal is not only to teach students knowledge but to shape their values, attitudes, and behaviors as they transition into their future careers.

Furthermore, DEIJ courses in undergraduate programs should be designed to foster inclusivity and engagement for all students, regardless of their backgrounds. This means offering multiple forms of participation, such as discussions, group projects, and reflective writing, to accommodate different learning styles and preferences. It also involves ensuring that students from historically underrepresented groups feel seen, heard, and valued in the classroom, which can contribute to a more equitable and supportive learning environment.

In essence, the data in this study both contradict an assumption many made about the supposed deficiency of online learning and document a truth that many educators have felt intuitively—that what youth lived through in the pandemic period made a difference in their worldviews. Importantly, in contrast to the fears of many that the traumatic aspects of the pandemic would leave humanity scarred or broken, these data paint a much more hopeful picture of human development in response to impactful world events. We are left with the question of what to do about the possibilities these findings reveal.