Next Article in Journal
Effectiveness of GBL in the Engagement, Motivation, and Satisfaction of 6th Grade Pupils: A Kahoot! Approach
Previous Article in Journal
A Critical Systematic Literature Review of Global Inclusive Education Using an Affective, Intersectional, Discursive, Emotive and Material Lens
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Education Sciences
Volume 13
Issue 12
10.3390/educsci13121213
education-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Evaluating Reflective Writing in Pre-Service Teachers: The Potential of a Mixed-Methods Approach

by
Chengming Zhang
*,
Jessica Schießl
,
Lea Plößl
,
Florian Hofmann
and
Michaela Gläser-Zikuda
Department of Education, Friedrich-Alexander-Universität Erlangen-Nürnberg, 90478 Nürnberg, Germany
*
Author to whom correspondence should be addressed.
Educ. Sci. 2023, 13(12), 1213; https://doi.org/10.3390/educsci13121213
Submission received: 16 November 2023 / Revised: 3 December 2023 / Accepted: 4 December 2023 / Published: 6 December 2023
(This article belongs to the Section Teacher Education)
Browse Figure
Versions Notes

Abstract

:
Reflective writing is a relevant aspect of pre-service teachers’ professionalization. Evaluating reflective writing in teacher education is demanding due to a shortage of resources. Hence, this study explores the practical possibilities of evaluating reflective writing using a mixed-methods approach to analyze reflective writing from 198 pre-service teachers at a German university. We used qualitative content analysis, computational linguistic approaches, and BERTopic. Results of qualitative content analysis results indicated primarily descriptive and low-level participants’ reflective writing. Next, computational linguistic analyses revealed that affective and cognitive terminology utilization differed across varying levels of reflection, with a higher frequency of such terms correlating with deeper levels of reflection. BERTopic results showed that reflective content mainly centered on learning materials and shifted toward affective and motivational themes related to higher levels of reflection. This study demonstrates that reflective writing can be evaluated across reflection levels and cognitive, affective, and thematic dimensions, combining qualitative content analysis, computational linguistic approaches, and BERTopic.

1. Introduction

The disconnection between theoretical understanding and practical application remains a long-standing challenge in teacher education [1]. For instance, novice teachers often have difficulty with classroom management, yet the relevant theoretical knowledge has already been learned in university, e.g., [2]. To bridge this gap, reflection has been increasingly recognized as essential in fostering professional development among (pre-service) teachers [2,3]. Reflection has been defined as a cognitive process that involves scrutinizing and evaluating experiences, situations, or thoughts [4,5,6], and it also includes critical perspectives, psychological activities, and affective dimensions [7]. In teacher education, there are various methods to support reflection, with portfolio-based reflective writing emerging as a predominant approach [8,9].
Despite the growing application of approaches to support reflection, there is evidence that teacher educators find it challenging to assess reflective writing [10]. On the one hand, the inherently subjective and complex nature of reflective writing complicates establishing universally applicable quality criteria [11]. This subjectivity makes it difficult to delineate what constitutes high-quality reflective writing. On the other hand, even when high-quality assessment models are available, their implementation demands substantial investments in time and resources—a luxury often unavailable to evaluators operating under constraints [12]. Given these challenges, there is a strong need for developing evaluation tools to support professionals in evaluating reflective writing. In current research, reflective writing assessment has traditionally been informed using content analysis, which is pivotal in evaluating complex aspects such as subjective experiences, emotions, and cognitive processes. For instance, the study by Poldner et al. [13] used quantitative content analysis to examine the various elements of reflective writing. This method provides measurable and comparable data for reliable and valid formative assessment of students’ reflective writing but is less efficient as well as not scalable. Moreover, as machine learning (ML) and natural language processing (NLP) technologies continue to advance, more automated approaches are emerging within the field. For example, Fan et al. [14] utilized Support Vector Machines; Kovanović et al. [15] employed Random Forest algorithms; Ullmann et al. [16] experimented with Neural Networks; and Wulff et al. [17] engaged pre-trained language models. These computational methods offer new avenues for evaluating reflective writing, potentially enhancing efficiency and scalability. However, these computational methods are not without limitations. Most notably, they often fall short of capturing the complex, contextual, and deeply personal facets of reflective writing.
While the existing literature provides valuable insights into approaches to evaluating reflective writing, this study attempts to explore the potential of a mixed-methods [18] approach in order to provide a more holistic understanding of reflective writing of pre-service teachers. Qualitative research methods help to explore subjective elements in depth to understand the motivation, knowledge, and situations behind reflective writing. Computational research methods provide the means to quantify these experiences, for example, by analyzing the frequency, length, and subject matter of writing, as well as linguistic features. Therefore, a mixed-methods approach seems suitable to capture these multifaceted dimensions, thus providing a more comprehensive analytical framework.

1.1. Definition of Reflection and Assessment Framework

Dewey’s book How We Think [4] was regarded as a landmark in the reflection area since it synthesized the ideas of many pioneers. John Dewey was the first to define reflection as a cognitive process, “an active, persistent, and careful consideration of any belief or supposed form of knowledge in the light of the grounds that support it and the further conclusions to which it tends” [4] (p. 9). Furthermore, reflection is described by Dewey [4] and Schön [5,6] as a primarily cognitive activity, notwithstanding the arguments of other scholars that it also entails critical, psychological, and affective components [19]. Mezirow defined reflection as “the process of critically assessing the content, process, or premise(s) of our efforts to interpret and give meaning to an experience” [20] (p. 104). Boud et al. [21] stated that the affective component of reflection is also essential since it may trigger specific experiences.
In orientation to these definitions, researchers have developed various assessment models of reflection in professional education [22]. These models reduce the complexity of reflection by representing it through frameworks [23], rubrics [24], or coding systems [13]. Most models focus on the depth or breadth of reflection for analyzing reflective writings [16]. The depth model frequently starts with a holistic evaluation of reflection [25]. For example, Hatton and Smith’s theory of reflection levels [26] is the most often used depth model in education, especially in teacher education. They proposed four levels of reflection, beginning with descriptive writing (only descriptive description) and progressing to descriptive reflection (providing justification based on the author’s opinion or relevant literature), dialogic reflection (setting up an inner dialogue to explain the possibilities), and critical reflection (considering the impact of one’s behavior in the larger historical, social, and political context before deciding what to do or why). Reflection was also conceptualized as a hierarchy model by Ip et al. [27], Kember [28], Mezirow [20], and others, ranging from non-reflective to highly reflective.
On the other hand, a multidimensional or process-oriented perspective on analyzing reflection was applied in breadth models [24]. For instance, Gibbs’s [29] reflective cycle consists of six parts that serve as a structure for creating and analyzing reflective writings: description (describe what happened briefly), feelings (describe feelings/emotional responses), evaluation (what was good/bad about response), analysis (how do you make sense of it), conclusions (general/specific conclusions), and action plan (what would you do next time). Furthermore, the breadth model has also been employed by Kolb [30], Mansvelder-Longayroux et al. [31], Poldner et al. [13], and others. These breadth models involve more elements but lack depth in critical thinking and analytical statements. This is because evaluating reflective writings contemplates their presence or absence as a criterion. Unlike the depth model, which explicitly takes the need to consider various perspectives objectively and the implication that learning will result in changes in assumptions, views, or practices, this model does not explicitly account for the assumptions people may have about the experience. Therefore, it is assumed that the levels of the student’s reflections would be more precisely identified and distinguished using a depth model. We adapted Hatton and Smith’s [26] theoretical framework to analyze pre-service teachers’ reflective writing depth.

1.2. Critical Role of Reflective Writing and the Challenges of Its Evaluation

A robust body of research shows the role of reflection in advancing (pre-service) teachers’ professional development [2,32,33]. Specifically, reflection enables both in-service and pre-service teachers to reconcile practical classroom experiences with theoretical pedagogical knowledge [3]. To facilitate this reflective practice, teacher education programs often employ portfolio-based strategies, e.g., [9,34,35]. However, reflection is a complex undertaking, particularly for novices [7]. Pre-service teachers tend to engage in reflection that is superficial and predominantly descriptive [36]. Körkkö et al. [37] similarly found that even when pre-service teachers’ reflections were broader and more thorough, they often lacked critical analytical depth. Poldner et al. [13] further emphasized that most reflections pivot around mere description and evaluation. They argue that genuine reflective practice should incorporate justification, dialogue, and transformative learning elements.
Consequently, for teacher educators, delivering effective assessments coupled with timely feedback constitutes a critical strategy for enhancing the quality of reflective writing. However, the assessment of such writing is laden with complexities and influenced by various factors that span cognitive, affective, technological, and individual assessor characteristics. First, the highly personal and subjective nature of reflective writing, often grounded in an individual’s experiences, emotions, and values, adds layers of complexity to its assessment, e.g., [19,22]. Second, practical constraints, notably in terms of time and resources, frequently hinder teacher educators from executing comprehensive and nuanced assessments, e.g., [12]. To mitigate these challenges, contemporary research has made strides in establishing clear, transparent assessment criteria [23,24], employing multidimensional assessment instruments [24], and leveraging technological advancements to facilitate the assessment process [16,38]. Nevertheless, to the best of our knowledge, there exists a research gap with respect to the evaluation of reflective writing in teacher education.

1.3. Multifaceted Approaches to the Evaluations of Reflective Writings

In teacher education, reflective writing is typically analyzed through two principal methodologies: manual analysis based on a theoretical framework and computational analysis with different technologies. The former mainly relies on content analysis, offering an in-depth interpretation of subjective experiences in reflective writing. The latter, conversely, frequently employs computational techniques that span dictionary-based, rule-based, and ML-based methods to perform the analysis.
First, most empirical research on how pre-service teachers reflect has been conducted through qualitative [39,40] and quantitative content analysis [41]. For example, Poldner et al. [13] used a quantitative content analysis based on theoretical coding schemes to analyze reflective writings of pre-service teachers over two semesters. This type of analysis is helpful for categorizing reflections since it seeks to answer whether or not reflection happens and to what extent. Similarly, Azimi et al. [36] coded 620 reflective writings from 41 pre-service teachers in three practicum cohorts over two years. Their study used a model based on Ward and McCotter [23] to demonstrate different reflective dimensions at different levels of reflection. Their findings focused not only on whether reflection occurs but also on how it occurs. These findings are relevant to describe characteristics, processes, and conditions of reflection from pre-service teachers’ views and to use these results to develop assumptions and theoretical frameworks. However, content analysis methods are limited to analyzing small samples and are time-consuming, making generalization difficult.
Another approach to analyzing reflective writing is computational analysis, which uses dictionaries to identify selected linguistic indicators or keywords associated with reflection. For instance, Springer and Yinger [38] used the Linguistic Inquiry and Word Count 2015 to extract keywords such as pronouns, temporal focus, and affect from reflective writing. This approach is useful for exploratory research but needs a theoretical foundation for assessing the quality of reflection. In contrast, Cui et al. [19] and Liu et al. [42] form a new reflective element by consolidating selected linguistic indicators into a unified taxonomy of reflection quality based on theoretical frameworks such as Gibbs [29] and Boud et al. [21]. However, a challenge in this approach is matching the linguistic indices with the reflective elements reasonably and progressively. Subsequently, there is a rule-based approach to assess reflective writing, but the number of empirical studies could be more significant. For example, Chong et al. [43] classified the level of reflection by building a word database with a fuzzy logic system due to a lack of training datasets and expertise. This fuzzy logic system is based on the keywords created by cognitive, emotive, and volitive words. It uses them as counts as inputs to the system to categorize the input text and determine the level of reflective practice. Gibson et al. [44] analyzed the text using low-level grammatical features (posTags) and higher-level annotations (metaTags) in reflective writing. The advantage of this approach is that it is more precise in analyzing the semantic aspects of reflective writing. However, it is difficult to generalize the analysis of reflective writing applied in different scenarios. Finally, ML approaches can be broadly categorized into supervised and unsupervised learning. In the context of reflective writing, supervised learning can be applied to text classification tasks, where the goal is to categorize texts into predefined categories, such as high or low levels of reflection. Researchers have employed various artificial intelligence (AI) algorithms and theoretical models to classify reflective texts [45]. For example, Liu et al. [42] used Naive Bayes to categorize reflective elements. On the other hand, unsupervised learning is a type of ML in which an algorithm learns to recognize patterns in data without explicitly using labeled data for training. For instance, Cutumisu and Guo [46] conducted a research study to analyze the reflective writing of 139 Canadian pre-service teachers using the Latent Dirichlet allocation (LDA) Topic Modeling. Their analysis revealed that pre-service teachers’ reflections included concepts, practices, and perspectives on computer thinking. Their findings highlight the potential of these methods to uncover hidden patterns and themes in large amounts of data and the benefits of using them to support reflective writing assessment in education. Therefore, a combination of different methods based on a mixed-methods approach seems advantageous, especially due to the complexity of phenomena in education [47], such as pre-service teachers’ reflective writing.

1.4. Research Questions

In light of the existing body of research, our study was formulated with three objectives. First, we aimed to evaluate the level of reflection based on Hatton and Smith’s framework [26]. This analysis served to establish a foundational understanding of the reflective capabilities exhibited by our participants. The second objective focused on employing LIWC2015 [48] and BERTopic [49] to extract terminology utilization and topics for characterizing the reflective writing. Both of these tools fall under the NLP and ML categories. Based on the above objectives, we formulated the following research questions:
RQ1: 
On which levels of reflection can reflective writings of pre-service teachers on profession-related topics be described and evaluated?
RQ2: 
How do the psycholinguistics of reflective writings differ among pre-service teachers at different levels of reflection?
RQ3: 
How do pre-service teachers at different levels of reflection differ in the topics of their reflective writings?

2. Materials and Methods

2.1. Research Context

We employed a mixed-methods approach [18,47] to synthesize the findings gleaned from the preceding objectives. Specifically, we sought to relate the linguistic features and reflective themes extracted from these writings with the levels of reflection derived from qualitative content analysis to gain a broader understanding of pre-service teachers’ reflective writing.
In conducting this study, an E-portfolio [50,51] was developed and integrated into an online learning platform as part of a teacher education program at a German university. This study was carried out during the winter semester of 2021/2022. This portfolio was designed to serve as a learning and reflection tool for two specific topics within a course on school education, namely, pedagogical diagnostics and classroom management. Next to the E-portfolio, pre-service teachers had access to various instructional resources, such as pre-recorded video lectures, presentation slides, and recommended readings. The portfolio contained case study assignments closely aligned with the topics covered to support the pre-service teachers’ understanding of the subjects and to initiate reflective writing. Specific assignments were designed to prompt pre-service teachers to reflect on the professional content acquired and their individual learning trajectories. The sequence of engagement with the portfolio was carefully structured. Initially, participants were mandated to engage with the learning materials of the course that were relevant to the topics in question. Upon completing these materials, they were then required to undertake a case study assignment directly pertinent to the learning topic. This sequential design ensured that pre-service teachers had basic knowledge before engaging in more complex tasks, reinforcing their learning and facilitating more in-depth understanding and reflection. Different time allocations were set for each of the two learning modules—pedagogical diagnostics and classroom management. Students were given three weeks to complete the pedagogical diagnostics module due to its complexity, whereas a week was allotted for the classroom management module. After completing the self-study components within these periods, students had two more weeks to submit a case assignment and a reflective writing.

2.2. Participants and Data Corpus

This study involved a diverse cohort of pre-service teachers enrolled in four distinct teacher education programs: primary school, lower secondary school, secondary school, and grammar school. Out of the initial pool, 506 participants completed the lecture series and the corresponding assignments, including reflective writings. For subsequent analysis, a random sampling technique was employed to select 200 reflective writings from the completed pedagogical diagnostics and classroom management assignments. Upon removing outliers, the final data corpus for analysis consisted of 198 reflective writings. The average word count for these reflective writings was 229.85, with a standard deviation of 148.99, and it ranged from a high of 806 in one writing to a low of 30 in another. In addition to the textual content, this study coded demographic background information of pre-service teachers associated with each of the 198 reflective writings. To collect personal information, we used anonymized codes to ensure compliance with the University’s data protection regulations. After matching the personal anonymity codes from the reflective writing with the ones from the questionnaire, 105 data sets were suitable for further analysis. Table 1 in this study provides a detailed demographic breakdown for these 105 pre-service teachers.

2.3. Data Analysis

First, the reflective writings were analyzed using a qualitative content analysis approach, as outlined by Mayring [52]. Specifically, this study adopted a theory-oriented qualitative structuring strategy of analysis based on the framework, initially established by Hatton and Smith [26], to categorize and interpret the data. The theory-oriented qualitative structuring strategy of analysis incorporated category definitions, anchor examples derived directly from the reflective writings, and coding rules.
In the second step, we used the dictionary-based LIWC method [48] to extract psycholinguistic features to characterize the reflective writings. The DE-LIWC 2015 [53] was applied in a German adaptation that formed more than 80 dictionary categories and contained 18,711 words, stems, and some emotive symbols. In our analysis, we extracted 73 empirically validated linguistic features from 198 reflective writings, and the remaining redundant features were removed. The extracted linguistic features were used to analyze variance (ANOVA) using the open-source statistical software R, version 4.1.2, [54] at different levels of reflection. We applied Yoon and Lai’s [55] subsampling technique since the reflective writings’ level sample had an uneven distribution.
The BERTopic architecture is a state-of-the-art topic modeling based on BERT word embedding, feature reduction clustering algorithms, and word representation to discover latent topics in reflective writings. The following analyses were carried out using Python 3.10 (i.e., https://www.python.org/, accessed on 10 October 2022). Meanwhile, the following different open-source libraries were used: NumPy, Pandas, SpaCy, Bertopic, PyTorch, scikit-learn, Matplotlib, and Seaborn. For analysis, firstly, we used a standard preparation pipeline, including tokenization, stop word removal, lemmatization, and N-gram extraction. Next, with the help of BERT embedding, dimensionality reduction, and clustering, the BERTopic topic modeling technique created dense clusters. Lastly, the class-based TF-IDF gave weights to the terms for each topic.

3. Results

3.1. Distribution of Reflection Levels

The results of the qualitative content analysis show that pre-service teachers’ reflections were primarily descriptive, with 37 students on the description level and 129 students on the descriptive reflection level (e.g., Table 2). Of the 198 pre-service teachers, only 31 were coded as persons on the level of dialogic reflection, and just one person was coded on the highest level, namely, critical reflection. In addition, there was no difference in the distribution of reflection levels regarding majors, with the main focus being descriptive reflection (nearly 70% of the total). To check the intercoder reliability, two coders independently analyzed selected reflective writings. Cohen’s Kappa was very good, with 0.97 for the topic of pedagogical diagnostics and 0.96 for classroom management.

3.2. Distribution of Reflection Levels

Starting with the measures under “linguistic style attributes,” we determined that the higher the level of reflection on the reflective writings, the more terms were associated with the word count and impersonal pronouns. A one-way ANOVA revealed that the effect of the word count on the reflection level was significant, F (1, 91) = 9.70, p = 0.002. The mean of impersonal pronouns’ occurrences in the three different reflection levels were 3.71, 4.52, and 4.71, respectively. As a result, the impersonal pronoun had a significant effect, F (1, 91) = 4.80, p = 0.031.
Additionally, we identified that terms related to adverbs and quantifiers were used more frequently at higher reflection levels. In contrast, the opposite was true for the article. There was a significant effect for an adverb, F (1, 91) = 19.97, p = 0.000, and a significant effect for quantifiers, F (1, 91) = 12.99, p = 0.000. Despite the fact that the article had a negative impact, F (1, 91) = 4.28, p = 0.041, higher levels of reflection were associated with fewer words.
Furthermore, we observed that affective processes, especially negative emotions, were more frequently used in the high reflective performance for the assessments of affective attributes and perception. All three features had a slight impact, Faffective-processes (1, 91) = 4.43, p = 0.038, Fnegative-emotion (1, 91) = 4.28, p = 0.029, and Ffeel (1, 91) = 4.91, p = 0.016, correspondingly.
Next, it was possible to distinguish between different levels of reflection using indicators by employing the linguistic features of the cognitive attributes. Indicators of cognitive process, such as discrepancy, certainty, differentiation, negations, and comparisons, were discovered in our data set to differentiate between different levels of reflective writing performance—the more relevant the cognitive attributes, the higher the level of reflection. Table 3 shows all the results.

3.3. Variations in Reflection Topics at Different Reflection Levels

Four groups were determined after the topic modeling procedure. Based on c-TF-IDF, Figure 1 displays the keywords for different topics (Topic 0: slide, tasks, find, feedback, read through, content, point, satisfied, detailed, fall; Topic 1: prepare, motivate, feel, learn, know, find, fun, literature, disruption, fall; Topic 2: teacher, slide, student, note, aspect, task, opinion, knowledge, satisfied, written; Topic 3: class, student, disruption, measure, teacher, react, behavior, teacher (female), wrong, strategy). The number of reflective writings in each category is presented in Table 4, and the reflective writings identified as topic −1 were classified as “noisy data”.
We carefully labeled each topic in Figure 1 based on the most relevant words. Most topics can be interpreted by considering their most representative terms. The number of themes in the most prominent part was 94, accounting for 47%. In topic 0, “learning materials and process”, participants reflected on the learning materials utilized during the learning process, such as slides, assignments, finding something, feedback, reading, and so on. Topic 1 was classified as “motivation and emotion” since it contained the phrases “motivate”, “feel”, and “fun” in its keywords. There were 31 total contributions to this topic, which amounted to 16%. The context of reflective writings was essential for the theme classifications of topic 2 and topic 3. Both topics centered on their respective scopes and were reflections on the course content. The representative terms identified topic 2 as “learning content (focus on pedagogical diagnostics)” and topic 3 as “learning content (focus on classroom management)”. Both topics comprised roughly 10% of the total, which was similarly reasonably similar. Finally, BERTopic could not identify relevant topics for roughly 20% of the reflective writings. Table 4. displays the number of each topic.
Table 5 shows how each theme was distributed among the varying levels of reflection. Surprisingly, the theme “motivation and emotion” appeared more frequently at higher levels of reflection, namely, 22.58% in dialogical reflection, 17.83% in descriptive reflection, and only 2.70% in the description. Another startling finding was that BERTopic failed to recognize more topics at lower reflection levels. For example, the dialogical reflection only makes up 10% of the total, whereas the description level makes up roughly 30%.

4. Discussion

The aim of this study was to analyze and evaluate the reflective writing skills of pre-service teachers employing a mixed-methods design. We integrated a qualitative content analysis, LIWC-based computational linguistic approach, and topic modeling using BERTopic to achieve this. On the one hand, this study provides a nuanced understanding of the quality and attributes of students’ reflective writing. On the other hand, it contributes to future development of a practical approach for evaluating them.
Firstly, regarding RQ1, the results of the qualitative content analysis indicated that most pre-service teachers’ reflective writing was at a low level, with almost none at the critical level. These findings are consistent with previous research on reflective writing by pre-service teachers [13]. Further research also shows that even at the end of teacher education programs, students’ reflective writings were still primarily descriptive and lacked critical thoughts [37]. The reasons for the low quality of reflection may be multifaceted. One potential factor may be that educational institutions do not adequately support reflective practice, e.g., [56]. Even though many countries have included reflection as a fundamental competency in their national teacher standards, curriculum frameworks for teacher education programs often emphasize acquiring specialized subject knowledge and practice competencies, with little focus on reflection [57]. Based on this, teacher educators may also fail to provide adequate scaffolding for reflective activities and constructive assessment and feedback to facilitate more profound reflection. On the pre-service students’ side, psychological barriers may also impede the quality of reflective writing [58]. Because reflective practice inherently involves self-critical assessment, low self-efficacy, apprehension about external criticism, or a general lack of confidence could deter students from engaging in meaningful self-reflection. Additionally, through several in-depth interviews, we noted that a significant number of pre-service teachers lacked an understanding of reflection. They did not even know how to reflect. This confluence of institutional, pedagogical, and psychological factors likely contributes to the observed limitations in the depth and quality of students’ reflective writing.
In addressing RQ2, we analyzed linguistic features in the reflective writings, and we found that affective and cognitive features may be used to predict the level of reflection. This result is in line with the dataset for Data Science and Pharmacy [59]. The level of reflection was connected to the affective characteristics underlying LIWC, which are denoted by the parameters LIWC.negemo and LIWC.affect. It is possible that affective characteristics may be interpreted as indicators of the level of reflection. For instance, negative words in pre-service teachers’ reflective writings may suggest that they are struggling with the content of reflection or their learning process and that they are reflecting on their emotions related to these challenges. Moreover, the affective characteristics of reflective writing can provide insights into how students connect their learning experiences, attitudes, and understanding to the content they reflect on. For example, if a student uses words such as “significant,” “influential,” or “relevant,” it may suggest that they are relating the content to their learning experience, which could indicate a higher level of reflection. On the other hand, one cannot determine the depth of the reflection based solely on the affective features. Instead of restricting these sentiments to merely descriptive language, deep critical contemplation should link them to shifts in one’s worldview [60]. In our study, we also found that cognition-related words were significant in predicting reflection levels. Cognitive characteristics refer to mental processes such as thinking, reasoning, problem-solving, and so on. Words associated with cognitive features, such as “analysis”, “synthesis”, “evaluation”, and “critical thinking”, can provide valuable insights into the level of cognitive engagement in their reflection. Moreover, when the lexicon of affective and cognition coexists in reflective writing, it often means that the writer encountered a confusing stimulus [6,61]. At the same time, the writer may dive into deep thought. This happens because of the mental process of reflection, which is used to make sense of things that are hard to understand [4]. The findings of our study show that when the amount of reflection is more significant, there is a corresponding increase in the frequency with which cognitive and affective linguistic markers appear. Several researchers have analyzed reflection using the LIWC index of cognitive processes and its sub-indicators, and they have proven the predictability of the results [19,62].
Finally, for RQ3, the pre-service teachers’ reflective writing topics were similar, demonstrating high homogeneity. Less than half of the students were thinking about the content they were currently working on, including the lecturers, slides, reading materials, and other matters. This is something that can be verified at every level of reflection. One possible explanation for this finding is the use of a specific prompt in our study. Most pre-service teachers were in their first university semester and may have lacked reflective writing experience, so most of them followed the assigned prompt in detail without creating their own reflection structure. The pre-service teachers may have experienced the reflective writings as part of an achievement assessment, although they did not get grades. Furthermore, learners’ motivation and academic success are positively correlated with their level of engagement with the learning content [63]. The learning content and the learning process are crucial factors in reflective writing. This is not surprising, given that reflective writing involves deep thinking and self-assessment of one’s understanding and application of the learning content. Moreover, the findings of this study demonstrated the various ways in which the theme of “motivation and emotion” may vary depending on the level of reflection. The more in-depth the reflection, the more material related to this issue emerges. This is consistent with the other results of our study, which show that the higher the percentage of affective and cognitive words in reflective writing, the higher the level of reflection. Affect is considered one of the components of reflection [64,65]. Moreover, it serves as a trigger that drives more profound reflection. The topic of reflection, “motivation and emotion”, in our study involved a change of the reflection perspective and, therefore, was found to be on a higher level of reflection.

4.1. Limitations and Future Directions

Some limitations of this study have to be addressed. Foremost among these is the issue of gender bias. Given that teacher education programs often have a high enrollment of female students, this study is similarly characterized by a gender imbalance as approximately 70 percent of the participants were female. It is essential to highlight that gender-specific linguistic features can manifest differently in reflective writing. Research indicates that females tend to employ linguistic elements that diverge from those utilized by males in their writing [62]. This gender imbalance could potentially skew the findings and interpretations, and it should be considered when evaluating the generalizability and applicability of this study’s outcomes. Additionally, constrained by resource limitations, the participant pool of the current study predominantly comprised first-year students. This demographic notably lacks substantial exposure to reflective training practices. Incorporating pre-service teachers from later years into this research could yield a broader array of results. Furthermore, using LIWC2015 as a pre-defined dictionary introduces additional limitations to the study. Its limited lexicon may not capture the entire vocabulary and linguistic nuances that could be present in reflective writing in this study that was carried out in a specific teacher education course in Germany. Moreover, LIWC2015 focuses on word frequency statistics without accounting for the semantic context in which these words are used. Consequently, the tool may miss or misinterpret the subtle complexities and interconnected meanings of terms within the reflections.
Future research directions might include methodological advances, technology integration, educational interventions, and so on. Firstly, in terms of methodological advances, it is imperative to aim for a more balanced data collection to enhance the scalability and generalizability of the research. This could involve considering variables such as gender, educational background, and other demographic factors influencing reflective writing. Additionally, the need for longitudinal studies is pressing, as understanding the development of reflective skills over time can offer deeper insights into their quality and influencing factors. Next, concerning technological integration, the current findings pave the way for incorporating more advanced AI technologies to analyze and assess reflective writing. For example, generative AI technologies could provide pre-service teachers with personalized and context-sensitive feedback, potentially elevating their reflective skills’ quality. In addition, the possibility that students’ acceptance of AI might affect the effectiveness of the feedback should also be considered [66].

4.2. Implications for Theory and Practice

This study has multiple implications for the theoretical understanding and practical application of evaluating pre-service teachers’ portfolio-based reflective writings. From a theoretical standpoint, this study enriches the multidimensional understanding of reflective writing analysis. It does so by probing into the relationship between reflection levels and various cognitive, affective, and thematic dimensions. This adds layers of complexity to the extant theoretical frameworks on reflective writing. In addition, focusing on specific cognitive and affective terms as quality indicators in reflective writing could have a meaningful impact on theories that define and measure “quality” in this context.
In terms of practical implications, based on a mixed-method approach, this study offers a first step for developing an assessment that is especially relevant in large-scale educational settings with resource constraints. Using NLP and ML techniques lays the foundation for developing automated tools capable of real-time assessment. This can significantly optimize the efficiency and effectiveness of the assessment process, which is a crucial consideration for educational institutions operating with limited resources. Secondly, the findings of this study offer valuable insights into the development of a design for feedback mechanisms for reflective writing. Such feedback could be strategically tailored to address the depth and breadth of reflection. Finally, this study’s observation that most reflective writing lacks quality and depth points to a pressing need to revisit and possibly revamp teacher training programs and potentially even higher education curricula more broadly. One plausible approach could be incorporating specific modules to cultivate reflective abilities within relevant courses. These could focus on teaching the theoretical aspects of reflection and the practical skills necessary for effective reflective writing.

5. Conclusions

Reflection is an essential concept bridging theory and practice, and it is integral to the professional development of teachers [1]. Nonetheless, one of the most pressing challenges in teacher education lies in pre-service teachers’ adequate evaluation of reflective writing [67,68]. Existing studies have predominantly relied on qualitative or quantitative content analysis, leaving a gap in covering larger samples and utilizing automated evaluation methods. Addressing this gap, our study employed a mixed-method approach, integrating diverse research methods to examine the relationship between the level of reflection, linguistic features, and the topics covered in reflective writing. This study’s principal findings revealed that pre-service teachers generally exhibited low levels of reflection. Further, it was determined that cognitive and affective linguistic features are crucial indicators for gauging the depth of reflection. Interestingly, this study also found a direct correlation between the level of reflection and the prevalence of motivational and emotional themes within reflective writing. The findings of this study carry considerable implications for assessing and analyzing reflective writing among pre-service teachers. They contribute to the existing body of research by revealing key indicators of reflection and point to the significance of reflective writing as an assessment tool for pre-service teachers’ professional development. Importantly, by incorporating a mixed-method approach, our study lays the groundwork for developing automated evaluation methods in this domain, streamlining the assessment process and enriching its analytical depth.

Author Contributions

Conceptualization, C.Z. and M.G.-Z.; Data curation, F.H.; Methodology, C.Z., J.S. and L.P.; Supervision, F.H. and M.G.-Z.; Writing—original draft, C.Z.; Writing—review & editing, J.S., L.P., F.H. and M.G.-Z.; Funding acquisition, M.G.-Z. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the German Federal Ministry of Education and Research under Grant 16DHB4019 obtained by Michaela Gläser-Zikuda.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

Due to the presence of personally identifiable information within the dataset, it is not publicly shareable in accordance with privacy protection laws and ethical guidelines of the universities Erlangen-Nürnberg and Berlin involved in this research.

Acknowledgments

We thank all members of the project for their support and ideas on this work. In particular, we thank our project partner Ralf Romeike, Christoph Benzmüller, Tim Landgraf, and Veronika Sopolova from FU Berlin, Germany.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Korthagen, F.A.J.; Kessels, J.P. Linking theory and practice: Changing the pedagogy of teacher education. Educ. Res. 1999, 28, 4–17. [Google Scholar] [CrossRef]
  2. Darling-Hammond, L.; Bransford, J. Preparing Teachers for a Changing World: What Teachers Should Learn and Be Able to Do; John Wiley & Sons: Hoboken, NJ, USA, 2007. [Google Scholar]
  3. Korthagen, F.A.J.; Kessels, J.; Koster, B.; Lagerwerf, B.; Wubbels, T. Linking Practice and Theory: The Pedagogy of Realistic Teacher Education; Routledge: London, UK, 2001. [Google Scholar]
  4. Dewey, J. How We Think; DigiCat: London, UK, 2022. [Google Scholar]
  5. Schön, D.A. The Reflective Practitioner: How Professionals Think in Action; Routledge: London, UK, 2017. [Google Scholar]
  6. Schön, D.A. Educating the Reflective Practitioner: Toward a New Design for Teaching and Learning in the Professions; Jossey-Bass: Hoboken, NJ, USA, 1987. [Google Scholar]
  7. Nguyen, Q.D.; Fernandez, N.; Karsenti, T.; Charlin, B. What Is Reflection? A Conceptual Analysis of Major Definitions and a Proposal of a Five-Component Model. Med. Educ. 2014, 48, 1176–1189. [Google Scholar] [CrossRef] [PubMed]
  8. Franco, R.S.; Franco, C.A.G.D.S.; Severo, M.; Ferreira, M.A.; Karnieli-Miller, O. Reflective Writing in the Teaching of Communication Skills for Medical Students—A Systematic Review. Patient Educ. Couns. 2022, 105, 1842–1851. [Google Scholar] [CrossRef] [PubMed]
  9. Gläser-Zikuda, M.; Feder, L.; Hofmann, F. Portfolioarbeit in der Lehrerinnen- und Lehrerbildung. In Handbuch Lehrerinnen- und Lehrerbildung; Cramer, C., König, J., Rothland, M., Blömeke, S., Eds.; Klinkhardt: Berlin, Germany, 2020; pp. 706–712. [Google Scholar]
  10. Ryan, M. The pedagogical balancing act: Teaching reflection in higher education. Teach. High. Educ. 2013, 18, 144–155. [Google Scholar] [CrossRef]
  11. Korthagen, F.A.J. Making Teacher Education Relevant for Practice: The Pedagogy of Realistic Teacher Education. Orb. Sch. 2018, 5, 31–50. [Google Scholar] [CrossRef]
  12. Ullmann, T. Automated Detection of Reflection in Texts: A Machine Learning Based Approach. Ph.D. Thesis, Open University, Milton Keynes, UK, 2015. [Google Scholar] [CrossRef]
  13. Poldner, E.; Van der Schaaf, M.; Simons, P.R.-J.; Van Tartwijk, J.; Wijngaards, G. Assessing Student Teachers’ Reflective Writing through Quantitative Content Analysis. Eur. J. Teach. Educ. 2014, 37, 348–373. [Google Scholar] [CrossRef]
  14. Fan, X.; Luo, W.; Menekse, M.; Litman, D.; Wang, J. Scaling reflection prompts in large classrooms via mobile interfaces and natural language processing. In Proceedings of the 22nd International Conference on Intelligent User Interfaces, Limassol, Cyprus, 13–16 March 2017. [Google Scholar] [CrossRef]
  15. Kovanović, V.; Joksimović, S.; Mirriahi, N.; Blaine, E.; Gašević, D.; Siemens, G.; Dawson, S. Understand Students’ Self-Reflections through Learning Analytics. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge, Sydney, NSW, Australia, 5–9 March 2018. [Google Scholar] [CrossRef]
  16. Ullmann, T.D. Automated Analysis of Reflection in Writing: Validating Machine Learning Approaches. Int. J. Artif. Intell. Educ. 2019, 29, 217–257. [Google Scholar] [CrossRef]
  17. Wulff, P.; Mientus, L.; Nowak, A.; Borowski, A. Utilizing a pretrained language model (BERT) to classify preservice physics teachers’ written reflections. Int. J. Artif. Intell. Educ. 2023, 33, 439–466. [Google Scholar] [CrossRef]
  18. Creswell, J.W. A Concise Introduction to Mixed Methods Research; SAGE Publications: Thousand Oaks, CA, USA, 2014. [Google Scholar]
  19. Cui, Y.; Wise, A.F.; Allen, K.L. Developing Reflection Analytics for Health Professions Education: A Multi-Dimensional Framework to Align Critical Concepts with Data Features. Comput. Hum. Behav. 2019, 100, 305–324. [Google Scholar] [CrossRef]
  20. Mezirow, J. Transformative Dimensions of Adult Learning; ERIC: New Haven, CT, USA, 1991. [Google Scholar]
  21. Boud, D.; Keogh, R.; Walker, D. Reflection: Turning Experience into Learning; Routledge: London, UK, 2013. [Google Scholar]
  22. Gore, J.M.; Zeichner, K.M. Action Research and Reflective Teaching in Preservice Teacher Education: A Case Study from the United States. Teach. Teach. Educ. 1991, 7, 119–136. [Google Scholar] [CrossRef]
  23. Ward, J.R.; McCotter, S.S. Reflection as a visible outcome for preservice teachers. Teach. Teach. Educ. 2004, 20, 243–257. [Google Scholar] [CrossRef]
  24. Miller-Kuhlmann, R.; O’Sullivan, P.S.; Aronson, L. Essential Steps in Developing Best Practices to Assess Reflective Skill: A Comparison of Two Rubrics. Med. Teach. 2015, 38, 75–81. [Google Scholar] [CrossRef] [PubMed]
  25. Jung, Y.; Wise, A.F. How and How Well Do Students Reflect? Multi-Dimensional Automated Reflection Assessment in Health Professions Education. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, Frankfurt, Germany, 23–27 March 2020; pp. 595–604. [Google Scholar] [CrossRef]
  26. Hatton, N.; Smith, D. Reflection in Teacher Education: Towards Definition and Implementation. Teach. Teach. Educ. 1995, 11, 33–49. [Google Scholar] [CrossRef]
  27. Ip, W.Y.; Lui, M.H.; Chien, W.T.; Lee, I.F.; Lam, L.W.; Lee, D. Promoting Self-Reflection in Clinical Practice among Chinese Nursing Undergraduates in Hong Kong. Contemp. Nurse 2012, 41, 253–262. [Google Scholar] [CrossRef] [PubMed]
  28. Kember, D. Determining the Level of Reflective Thinking from Students’ Written Journals Using a Coding Scheme Based on the Work of Mezirow. Int. J. Lifelong Educ. 1999, 18, 18–30. [Google Scholar] [CrossRef]
  29. Gibbs, G. Learning by Doing: A Guide to Teaching and Learning Methods; Further Education Unit: Dublin, Ireland, 1988. [Google Scholar]
  30. Kolb, D.A. Experiential Learning: Experience as the Source of Learning and Development; FT Press: Upper Saddle River, NJ, USA, 2014. [Google Scholar]
  31. Mansvelder-Longayroux, D.D.; Beijaard, D.; Verloop, N. The Portfolio as a Tool for Stimulating Reflection by Student Teachers. Teach. Teach. Educ. 2007, 23, 47–62. [Google Scholar] [CrossRef]
  32. Baumert, J.; Kunter, M. Stichwort: Professionelle Kompetenz von Lehrkräften. Z. Erziehwiss. 2006, 9, 469–520. [Google Scholar] [CrossRef]
  33. van Beveren, L.; Roets, G.; Buysse, A.; Rutten, K. We all reflect, but why? A systematic review of the purposes of reflection in higher education in social and behavioral sciences. Educ. Res. Rev. 2018, 24, 1–9. [Google Scholar] [CrossRef]
  34. Trent, J.; Shroff, R.H. Technology, Identity, and Community: The Role of Electronic Teaching Portfolios in Becoming a Teacher. Technol. Pedagog. Inf. 2012, 22, 3–20. [Google Scholar] [CrossRef]
  35. Xerri, D.; Campbell, C. E-portfolios in teacher development: The better option? ELT J. 2016, 70, 392–400. [Google Scholar] [CrossRef]
  36. Azimi, E.; Kuusisto, E.; Tirri, K.; Hatami, J. How Do Student Teachers Reflect on Their Practice through Practicum Courses? A Case Study from Iran. J. Educ. Teach. 2019, 45, 277–289. [Google Scholar] [CrossRef]
  37. Körkkö, M.; Kyrö-Ämmälä, O.; Turunen, T. Professional Development through Reflection in Teacher Education. Teach. Teach. Educ. 2016, 55, 198–206. [Google Scholar] [CrossRef]
  38. Springer, D.G.; Yinger, O.S. Linguistic Indicators of Reflective Practice among Music Education Majors. J. Music Teach. Educ. 2018, 28, 56–69. [Google Scholar] [CrossRef]
  39. Lee, S.J.C.; Abdul Rabu, S.N. Google Docs for Higher Education: Evaluating Online Interaction and Reflective Writing Using Content Analysis Approach. Educ. Inf. Technol. 2021, 27, 3651–3681. [Google Scholar] [CrossRef]
  40. Alsina, Á.; Ayllón, S.; Colomer, J. Validating the Narrative Reflection Assessment Rubric (NARRA) for Reflective Narratives in Higher Education. Assess. Eval. High. Educ. 2018, 44, 155–168. [Google Scholar] [CrossRef]
  41. Houston, C.R. Do Scaffolding Tools Improve Reflective Writing in Professional Portfolios? A Content Analysis of Reflective Writing in an Advanced Preparation Program. Action Teach. Educ. 2016, 38, 399–409. [Google Scholar] [CrossRef]
  42. Liu, M.; Shum, S.B.; Mantzourani, E.; Lucas, C. Evaluating Machine Learning Approaches to Classify Pharmacy Students’ Reflective Statements. In Artificial Intelligence in Education: 20th International Conference, AIED 2019, Proceedings, Part I; Springer International Publishing: Chicago, IL, USA, 2019; Volume 20, pp. 220–230. [Google Scholar] [CrossRef]
  43. Chong, C.; Sheikh, U.U.; Samah, N.A.; Sha’ameri, A.Z. Analysis on Reflective Writing Using Natural Language Processing and Sentiment Analysis. IOP Conf. Ser. Mater. Sci. Eng. 2020, 884, 012069. [Google Scholar] [CrossRef]
  44. Gibson, A.; Kitto, K.; Bruza, P. Towards the Discovery of Learner Metacognition from Reflective Writing. J. Learn. Anal. 2016, 3, 22–36. [Google Scholar] [CrossRef]
  45. Solopova, V.; Rostom, E.; Cremer, F.; Gruszczynski, A.; Witte, S.; Zhang, C.; López, F.R.; Plößl, L.; Hofmann, F.; Romeike, R.; et al. PapagAI: Automated Feedback for Reflective Essays. In German Conference on Artificial Intelligence (Künstliche Intelligenz), Berlin, Germany, 18 September 2023; Springer Nature: Cham, Switzerland, 2023; pp. 198–206. [Google Scholar] [CrossRef]
  46. Cutumisu, M.; Guo, Q. Using Topic Modeling to Extract Pre-Service Teachers’ Understandings of Computational Thinking from Their Coding Reflections. IEEE Trans. Educ. 2019, 62, 325–332. [Google Scholar] [CrossRef]
  47. Mejeh, M.; Hagenauer, G.; Gläser-Zikuda, M. Mixed Methods Research on Learning and Instruction—Meeting the Challenges of Multiple Perspectives and Levels Within a Complex Field. Forum Qual. Soc. Res. 2023, 24, 14. [Google Scholar] [CrossRef]
  48. Pennebaker, J.W.; Boyd, R.L.; Jordan, K.; Blackburn, K. The Development and Psychometric Properties of LIWC2015; University of Texas at Austin: Austin, TX, USA, 2015; Available online: http://hdl.handle.net/2152/31333 (accessed on 10 October 2022).
  49. Grootendorst, M. BERTopic: Neural Topic Modeling with a Class-Based TF-IDF Procedure. arXiv 2022, arXiv:2203.05794. Available online: http://arxiv.org/pdf/2203.05794v1 (accessed on 10 October 2022).
  50. Gläser-Zikuda, M. ePortfolios in Higher Education. In Encyclopedia of Educational Technology; Spector, M., Ed.; SAGE: Thousand Oaks, CA, USA, 2015; pp. 275–277. [Google Scholar]
  51. Gläser-Zikuda, M. Self-reflecting Methods of Learning Research. In Encyclopedia of the Sciences of Learning; Seel, N., Ed.; Springer: Berlin/Heidelberg, Germany, 2012; pp. 3011–3015. [Google Scholar]
  52. Mayring, P. Qualitative Content Analysis: Theoretical Foundation, Basic Procedures and Software Solution. Klagenfurt 2014. Available online: https://nbn-resolving.org/urn:nbn:de:0168-ssoar-395173 (accessed on 23 January 2023).
  53. Meier, T.; Boyd, R.L.; Pennebaker, J.W.; Mehl, M.R.; Martin, M.; Wolf, M.; Horn, A.B. “LIWC auf Deutsch”: The development, psychometrics, and introduction of DE-LIWC2015. PsyArXiv 2019. Available online: https://osf.io/tfqzc/ (accessed on 10 October 2022).
  54. R Core Team. R: A Language and Environment for Statistical Computing; The R Project for Statistical Computing; R Foundation for Statistical Computing: Vienna, Austria, 2010; Available online: https://www.r-project.org/ (accessed on 10 October 2022).
  55. Yoon, M.; Lai, M.H.C. Testing Factorial Invariance with Unbalanced Samples. Struct. Equ. Model. 2017, 25, 201–213. [Google Scholar] [CrossRef]
  56. Westbury, I.; Hopmann, S.; Riquarts, K. Teaching as a Reflective Practice: The German Didaktik Tradition; Routledge: London, UK, 2012. [Google Scholar]
  57. Batra, P. Voice and Agency of Teachers: Missing Link in National Curriculum Framework 2005. Econ. Political Wkly. 2005, 40, 4347–4356. [Google Scholar]
  58. Ramlal, A.; Augustin, D.S. Engaging Students in Reflective Writing: An Action Research Project. Educ. Action Res. 2019, 28, 518–533. [Google Scholar] [CrossRef]
  59. Liu, M.; Kitto, K.; Buckingham Shum, S. Combining Factor Analysis with Writing Analytics for the Formative Assessment of Written Reflection. Comput. Hum. Behav. 2021, 120, 106733. [Google Scholar] [CrossRef]
  60. Lucas, C.; Smith, L.; Lonie, J.M.; Hough, M.; Rogers, K.; Mantzourani, E. Can a Reflective Rubric Be Applied Consistently with Raters Globally? A Study across Three Countries. Curr. Pharm. Teach. Learn. 2019, 11, 987–994. [Google Scholar] [CrossRef] [PubMed]
  61. Cushion, C.J. Reflection and Reflective Practice Discourses in Coaching: A Critical Analysis. Sport Educ. Soc. 2018, 23, 82–94. [Google Scholar] [CrossRef]
  62. Lin, C.-W.; Lin, M.-J.; Wen, C.-C.; Chu, S.-Y. A Word-Count Approach to Analyze Linguistic Patterns in the Reflective Writings of Medical Students. Med. Educ. Online 2016, 21, 29522. [Google Scholar] [CrossRef]
  63. Nwankwo, A.A. Students’ Learning Experiences and Perceptions of Online Course Content and Interactions. Ph.D. Thesis, Walden University, Minneapolis, MN, USA, 2015. Available online: https://www.proquest.com/openview/e81b93ab0f7baf422ed2b10c367bddad/1?pq-origsite=gscholar&cbl=18750 (accessed on 23 February 2023).
  64. Cleary, M.; Horsfall, J.; Happell, B.; Hunt, G.E. Reflective Components in Undergraduate Mental Health Nursing Curricula: Some Issues for Consideration. Issues Ment. Health Nurs. 2013, 34, 69–74. [Google Scholar] [CrossRef]
  65. Gentile, D.L. Applying the Novice-To-Expert Model to Infusion Nursing. J. Infus. Nurs. 2012, 35, 101–107. [Google Scholar] [CrossRef]
  66. Zhang, C.; Schießl, J.; Plößl, L.; Hofmann, F.; Gläser-Zikuda, M. Acceptance of artificial intelligence among pre-service teachers: A multigroup analysis. Int. J. Educ. Technol. High Educ. 2023, 20, 49. [Google Scholar] [CrossRef]
  67. Embo, M.P.C.; Driessen, E.; Valcke, M.; Van Der Vleuten, C.P.M. Scaffolding Reflective Learning in Clinical Practice: A Comparison of Two Types of Reflective Activities. Med. Teach. 2014, 36, 602–607. [Google Scholar] [CrossRef] [PubMed]
  68. Sargeant, J.; Lockyer, J.; Mann, K.; Holmboe, E.; Silver, I.; Armson, H.; Driessen, E.; MacLeod, T.; Yen, W.; Ross, K. Facilitated reflective performance feedback: Developing an evidence-and theory-based model that builds relationship, explores reactions and content, and coaches for performance change (R2C2). Acad. Med. 2015, 90, 1698–1706. [Google Scholar] [CrossRef] [PubMed]
Education 13 01213 g001
Figure 1. List of related words based on c-TF-IDF.
Figure 1. List of related words based on c-TF-IDF.
Education 13 01213 g001
Table 1. Pre-service teachers’ (n = 105) demographic information.
Table 1. Pre-service teachers’ (n = 105) demographic information.
VariableItemn
GenderMale35
Female70
MajorPrimary school (Grundschule)55
Lower secondary school (Mittelschule)15
Secondary school (Realschule)12
Grammar school (Gymnasium)25
Age(M/SD)21.63/4.01
Semester(M/SD)2.00/1.51
Note. Ninety-three instances of data loss occurred because of personal code-matching issues.
Table 2. Distribution of pre-service teachers to the reflection levels by learning topics.
Table 2. Distribution of pre-service teachers to the reflection levels by learning topics.
DescriptionDescriptive ReflectionDialogic ReflectionCritical Reflection
Pedagogical diagnostics2061181
Classroom management1768130
Total37129311
Table 3. ANOVA comparing the psycholinguistic attributes of reflective writing at different reflection levels.
Table 3. ANOVA comparing the psycholinguistic attributes of reflective writing at different reflection levels.
Linguistic CategoryDescriptionDescriptive ReflectionDialogical Reflection
M/SDM/SDM/SDF
Linguistic Style Attributes
Word Count177.97/79.47261.26/149.5274.87/126.579.70 **
Impersonal Pronouns3.71/1.344.52/2.094.71/1.874.80 *
Lexical Density
Article13.20/2.7111.08/2.0810.98/2.134.28 *
Adverb4.15/1.745.28/1.926.19/1.7719.97 ***
Quantifiers2.31/1.153.39/1.223.62/1.8212.99 ***
Affective Attributes
Affective processes5.04/2.325.79/2.476.21/1.744.43 *
Negative emotion1.06/0.771.30/1.061.63/1.174.91 *
Perception
Feel0.23/0.410.51/0.720.62/0.696.08 *
Cognitive Attributes
Cognitive processes21.05/3.9723.12/4.2125.65/3.4421.92 ***
Discrepancy1.88/1.182.51/1.403.38/1.4719.25 ***
Certainty3.09/2.272.95/1.464.16/1.884.89 *
Differentiation3.69/1.994.19/1.725.83/2.1917.93 ***
Negations0.72/0.920.86/0.801.61/1.0613.70 ***
Comparisons2.08/1.222.67/1.032.97/1.109.71 **
Note. Statistical significance was reported after subsampling for class imbalance; each group consisted of 31 samples (*** p < 0.001, ** 0.001 < p < 0.01, * 0.01 < p < 0.05).
Table 4. Topic counts computed with BERTopic and their proportional distributions over the entire dataset.
Table 4. Topic counts computed with BERTopic and their proportional distributions over the entire dataset.
TopicCount%
Topic 0: Learning materials and process9447
Topic 1: Motivation and emotion3116
Topic 2: Learning content (focus on pedagogical diagnostics)2010
Topic 3: Learning content (focus on classroom management)158
Topic—1: noisy data3819
Table 5. Distribution of topic counts in different reflection levels.
Table 5. Distribution of topic counts in different reflection levels.
TopicDescription
(n = 37)		Descriptive Reflection
(n = 129)		Dialogical Reflection
(n = 31)
Learning materials and process186115
Motivation and emotion1237
Learning content
  focus on pedagogical diagnostics5123
  focus on classroom management383
Noisy data10253
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Zhang, C.; Schießl, J.; Plößl, L.; Hofmann, F.; Gläser-Zikuda, M. Evaluating Reflective Writing in Pre-Service Teachers: The Potential of a Mixed-Methods Approach. Educ. Sci. 2023, 13, 1213. https://doi.org/10.3390/educsci13121213

AMA Style

Zhang C, Schießl J, Plößl L, Hofmann F, Gläser-Zikuda M. Evaluating Reflective Writing in Pre-Service Teachers: The Potential of a Mixed-Methods Approach. Education Sciences. 2023; 13(12):1213. https://doi.org/10.3390/educsci13121213

Chicago/Turabian Style

Zhang, Chengming, Jessica Schießl, Lea Plößl, Florian Hofmann, and Michaela Gläser-Zikuda. 2023. "Evaluating Reflective Writing in Pre-Service Teachers: The Potential of a Mixed-Methods Approach" Education Sciences 13, no. 12: 1213. https://doi.org/10.3390/educsci13121213

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop