Diagnostic and Feedback Behavior of German Pre-Service Teachers Regarding Argumentative Pupils’ Texts in Geography Education

Abstract

The study sheds light on the importance of diagnostic and feedback skills regarding argumentative pupils’ texts for (prospective) teachers and the current state of research in this area. The importance of argumentation in geography education is undisputed, as geographical problems often require multi-layered solutions that can be analyzed, assessed and reflected upon by teachers and pupils through argumentation skills. Nevertheless, research findings reveal that both teachers and pupils often have limited argumentation skills, which poses a challenge for both. The present study investigates the attitudes and practices of prospective geography teachers regarding the diagnosis of and feedback on pupils’ argumentative texts. Using a qualitative research design, twenty German pre-service teachers of geography were interviewed about their diagnostic and feedback experiences and asked to analyze and provide feedback on an argumentative pupil text. The results were evaluated using qualitative content analysis. In addition, a theoretical model for diagnosis was developed, which was used to analyze the pre-service teachers’ diagnostic behavior. The criteria used by the pre-service teachers to evaluate the argumentative pupils’ text were examined. The results show that the diagnosis often lacks the exploration of causes and the deduction of consequences, and that only a few subject-specific criteria were taken into account in the text assessment. In addition, the feedback was generally more positive than the diagnosis itself. The results offer insights into the design of diagnostic and feedback processes in geography education and provide impulses for (higher) education teaching and future research.

1. Introduction

It is no secret that argumentation is considered relevant in geography education: many geography didactic studies deal with the significance, scope and practical application of argumentation and the quality and promotion of argumentation skills of pupils, students and teachers in geography education [1,2]. The relevance results from the content and the role of the subject: there is rarely a clear-cut solution to geographical problems, as a wide range of factors and actors exert an influence. Examples of this are discourses on conflicts over the use of space, climate change or border conflicts. Argumentation can help to assess and evaluate conflicts, to position oneself accordingly and to penetrate the subject matter. It is also a means of discussing solutions to problems [3] and finding compromises [4]. It is, therefore, an important skill that should be taught to pupils in geography lessons. However, research results show that many students have only limited argumentation skills, both in writing and orally, and that argumentation tasks and their demands represent a challenge for pupils and teachers [1,2,5,6,7,8]. It follows from this that argumentation skills should be promoted more strongly in geography education and that teachers’ insecurities and challenges with regard to argumentative tasks must be addressed. Support can only start where skills are diagnosed and feedback is provided. As Hußmann and Selter [9] described, “Support without prior diagnosis is usually unspecific, diagnosis without subsequent support is often ineffective and often leads to stigmatization” (p. 16). Teachers must, therefore, be supported in evaluating pupils’ reasoning and diagnosing their skills in order to subsequently give them qualified and comprehensible feedback and support them.

Although good diagnostic and feedback skills of teachers with regard to argumentative pupil texts are very important and form the basis for didactic support, studies have not yet focused on this area. Qualitative study results only show in the area of the assessment of an argumentative pupil text by geography teachers, which is the basis for diagnosis and feedback, that linguistic criteria are largely assessed and corrected in the subject context instead of subject-specific criteria [10]. Furthermore, teachers describe that they only acquired their relevant skills through learning-by-doing and not during their studies (ibid.). This makes it all the more relevant to prepare student teachers for diagnosis and support during their university education. In order to be able to start concrete implementation of support in university didactics, it is necessary to research the current status of pre-service teachers and their current practices, as well as their attitudes towards diagnosis and feedback on argumentative pupil texts. The following study, therefore, examines prospective geography teachers in the Bachelor’s degree program. It focuses on the attitudes and practices of German pre-service geography teachers with regard to diagnosis and feedback on argumentative pupil texts. The following research questions are examined:

• What experiences and attitudes do pre-service geography teachers have towards diagnosing pupils’ argumentative competencies in geography education?
• How do pre-service geography teachers practice diagnosing pupils’ argumentative competence based on their argumentative texts?
• How do pre-service geography teachers give feedback on pupils’ argumentative texts?

To answer the research questions, twenty pre-service geography teachers were interviewed about their attitudes as part of a subject-didactic bachelor seminar using a qualitative research design with open questions. They were then asked to correct and evaluate an argumentative pupil text, diagnose the pupils’ abilities and formulate feedback. The research design is a qualitative experimental setting, the initial aim of which is to gather first impressions on the previously unexplored topic of diagnosis and feedback on argumentative texts in the context of geography education. The research questions will be answered in the subsequent evaluation through qualitative content analysis based on defined deductive diagnostic and feedback criteria.

The following section first provides an insight into the definitions and relevance of written argumentation skills for pupils as well as the diagnostic and feedback skills of pre-service geography teachers and explains a theoretical model (based on Aufschnaiter, Münster and Beretz, 2018:384) [11] for a successful diagnostic cycle. This is followed by an overview of the current state of interdisciplinary research on the diagnosis of pupil competencies and teacher feedback. As a result, the research methodology is explained and quality criteria for diagnosis and feedback on argumentative pupil texts are listed. Finally, the results of the study are presented and discussed and an outlook for (higher) education teaching and research is given.

2. Theory

2.1. Written Argumentation (Skills) in Geography Education

Geography education is about understanding cause-and-effect relationships (e.g., human impact on nature), interpreting tensions (e.g., conflicts over the use of space), formulating and testing hypotheses about geographical content, comparing facts using different criteria (e.g., spatial disparities), arguing about geographical concepts and forming one’s own opinions. It is, therefore, about dealing with problems, and argumentation can help to solve them [12].

In this way, geography education can develop learners’ critical thinking and reasoning skills to make well-reasoned decisions on social and environmental issues [13]. Incorporating argumentation as a teaching method in geography improves pupils’ understanding of key geographical concepts while contributing to their development as responsible citizens [14]. These are just some of the promising functions that justify argumentation in geography education. However, in order to fulfil them, pupils need argumentation skills that should be demanded by geography teachers.

Budke [15] defines argumentation competence, based on Weinert’s [16] concept of competence, as the ability and skills to receive, produce and interact with oral and written arguments. For this purpose, pupils must have the social, volitional and motivational readiness to use their argumentation skills successfully and responsibly in different situations [15]. Furthermore, based on the Common European Framework of Reference (CEFR), a model for the development of individual areas of argumentation skills, was developed, which distinguishes in particular between oral and written argumentation skills at the reception/interaction and production levels [17]. According to this, pupils should ‘take a position on geographical issues and justify it in writing’ (ibid.).

A particular requirement for arguments in the geographical sense is that they are open-ended [18] and that an opinion is formed only in the course of an argument, that they have a spatial reference and different perspectives, and that they should be justified in a complex way [19]. These requirements exist because geography education deals with controversial topics and perspectives of social relevance, and space is seen as a focus of the subject (ibid).

Despite this, previous research has shown that argumentation skills are difficult to teach due to the range of different criteria that need to be considered [20,21,22]. They were seen as too time- and space-consuming for subject teaching and, therefore, also perceived as a resource issue. Furthermore, argumentation is challenging for both pupils and teachers due to its high level of complexity; studies across all age groups have shown that both pupils and teachers perform poorly in producing and evaluating their own arguments [23,24,25,26].

In order to summarize the complexity and to sensitize geography teachers to the different requirements of argumentative pupils’ texts and to provide them with an overview of assessment criteria, a theory-based criteria grid for the assessment of argumentative pupils’ texts in geography lessons was developed [10]. It is divided into three main areas: Language, Argumentation and Structure. Analysis criteria are assigned to these sub-areas and explained using text examples. Language is defined as a large sub-area because without appropriate language, successful communication about the subject matter cannot take place, and these general language criteria exist across all subjects and are considered a basic requirement (ibid.). Subject-specific language criteria, such as the correct use of technical terminology, apply particularly to geography teaching [27,28,29,30].

The analysis area of argumentation also deals with the quality and quantity of the arguments mentioned in the pupils’ texts and is important because the analysis of the understanding of the content of the subject and the linking of geographical contexts, in particular, are analyzed. In geography education, these criteria are primarily the presentation of controversies [31], complexity [2,28], spatial and temporal conditions under which the argument is valid [2] and factual relevance [6,32].

The third major area of analysis is the text structure, which is relevant for the generation of technical connections and syntheses between the individual arguments and the actor positions contained therein. From a structural point of view, the text type [33] and its organisation into an introduction, main body and conclusion [30] must be taken into account for the argumentative text in geography lessons.

In addition, there are general criteria that are also interdisciplinary (expression, spelling, valid evidence, pupil positioning, etc.). The high complexity of the sub-competences required in argumentation tasks makes a differentiated diagnosis necessary [30]. What diagnosis (competencies) means for teachers is explained in more detail below.

2.2. Diagnostic (Skills)

Diagnosis can be understood as the collection and evaluation of information about a situation or circumstance in order to form an overall judgement [34]. Diagnostics carried out at school always serve to optimize learning processes and form the starting point for support measures to be taken and internal differentiation [35,36].

For the German education system, the relevance of diagnostics is derived from the national educational standards for the subject of geography and results from the requirement to systematically record and promote pupils’ competencies [37]. Diagnostic skills are particularly relevant when it comes to teaching adaptively and designing individual support measures for pupils [38]. Diagnostics should enable teachers to obtain information about the level of competence of the pupils being taught in order to make decisions about teaching practice and support on this basis. In addition, appropriate grades should be given and parents and colleagues should be informed about performance [39]. Without individualized support, it is, therefore, not possible to optimize learning and promote skills [40]. Walz and Roth [41] also emphasize the relevance of diagnostic skills for teachers as a prerequisite “to be able to carry out suitable real-time interventions” (p. 1368). They are also useful for uncovering pupils’ misconceptions and guiding reflection on learning processes. Following Weinert’s definition [16], diagnostic competence is precisely the cognitive abilities and skills of a teacher as well as their affective-motivational readiness, which is required to fulfill diagnostic tasks in the context of teaching and learning. Helmke [42] emphasizes, however, that diagnostics serve in particular to gather information with the aim of making individualized decisions for individual support and adjustments to the teaching offer. Diagnostics are initially value-free.

Thus, diagnostics (a) focus on solutions and editing processes in order to make statements about pupil competencies [42], (b) focus on competence requirements in order to check the extent to which the lesson content and the teacher’s requirements match pupil levels [40] and (c) focus on follow-up support [43].

Therefore, diagnostic competencies are extensive and require professional competence, didactic competence [44], classroom management competence [45], analytical competence [46] and linguistic competence. It must be possible to apply subject-specific skills and then reflect on them. Teachers who are well trained in this respect were proven to lead to better pupil performance (ibid.). Studies show that teachers’ accuracy of judgment correlates with pupils’ learning success and that diagnostic skills are positively related to the quality of teaching [35]. Hattie [47] adds to this in his study, demonstrating the positive influence of regular feedback on learning success.

Steingrübl and Budke [10] use argumentative pupil texts to illustrate the typical steps of diagnostic teacher behavior: Teachers need to have different competencies that build on each other in order to successfully promote pupils’ competences (see Figure 1). The starting point is correction and text evaluation. This is used to make initial observations about sources of error in the text and then to make statements about the pupils’ competencies, which in turn is part of the diagnosis. Diagnosis is the key competence here and is, therefore, part of the professional behavior of every teacher and should, therefore, meet certain standards: it should be systematic and based on scientific criteria and concepts, focused on the task at hand and at the same time based on the reflective experience of the teacher making the assessment [39,48]. Following the diagnosis, feedback is given to the pupil by explaining the similarities and differences between the teacher’s expectations and the pupil’s performance. Feedback also provides an opportunity to communicate with the learner about the existing deficits and then to identify the causes and agree on joint learning objectives for subsequent support. The feedback step is, therefore, essential for subsequent support based on correction, diagnosis and feedback [10].

Now that the terminology and relevance of diagnostics in the classroom are clarified, the following section will focus on the diagnostics of pupils’ written argumentation skills in geography lessons.

2.3. Diagnostic of Pupils’ Written Argumentative Skills in Geography Lessons

In addition to interdisciplinary principles, diagnostic procedures in geography education also include didactic considerations: These provide criteria and guiding questions for diagnosis, which can already refer to aspects relevant for support and allow categorisation of interpretations regarding the causes of the pupils’ competence level and can be used to derive consequences in the form of support measures [49]. Subject-specific criteria that apply to the diagnostic of written argumentation skills have already been presented in Section 2.1. Beretz [49] has also developed a model for support-oriented diagnostics for (subject-specific) didactic theories and concepts based on the considerations of von Aufschnaiter, Münster and Beretz [11] as part of her research into the diagnostic skills of specialist teachers in physics and mathematics lessons. Diagnostic components based on (subject-specific) didactic considerations are summarized here. The scientific experiment is alienated as the basis for the considerations. In the following, the model (Figure 2) is expanded and specified for the diagnostic methods in geography education regarding argumentative texts and serves to evaluate the study results and to guide geography teachers in diagnosing pupils’ argumentative writing skills in geography lessons.

Figure 2 shows the five relevant, consecutive steps for a successful diagnostic of pupils’ written skills. Before the first step shown here, learners are usually given a task in which, for example, the expected type of text and the materials to be used are specified. This can be followed, for example, by the pupils’ text production, which is the starting point for the first step of the diagnostic cycle.

1.

Recording performance

An initial basis for diagnostic considerations and questions is the inspection of pupils’ products. This involves recording the pupils’ performance. This provides teachers with an initial orientation in terms of existing pupil competencies and a basis for further diagnostic steps. It is important to separate descriptions of performance and interpretations of competencies. Kohlhauf et al. [50] described the challenge teachers face in separating a factual description from the assessment. Clearly defined assessment criteria and a preceding text correction and assessment are particularly suitable for this in order to objectively determine the pupils’ deficits and competencies. A value-free description of the observations relevant to support should be available for this purpose. This can also have the advantage that pupils, other teachers or parents can understand the basis for their subsequent interpretations of the pupils’ skills and that there is room for other interpretations. The precise formulation of the observations forms the prerequisite for the skills analysis [49].

• What does this mean in relation to argumentative pupil texts?

The recording of performance in relation to argumentative pupil products is the first text review. Teachers gain an initial impression and can make corrections to the text. By definition, a text correction does not yet include an evaluation, as this is merely an “improvement; correction; rectification” [51] of text-immanent errors. The first problem areas [52] and error patterns [53] can already be clarified here. In a study on the correction and text assessment behavior of teachers, test subjects report that they make corrections, especially in weaker texts, in order to make the text readable [10].

2.

Analyzing and interpreting competencies

In the second diagnostic step, pupils’ skills are analyzed and then interpreted for feedback. This enables a classification and is relevant in order to identify the pupils’ understanding of the content and, if necessary, to adapt the lessons accordingly. The pupil’s product and the descriptions of the performance from the first step are analyzed, evaluated and interpreted against the background of selected tasks, materials and criteria defined by the teacher. Here, recourse to subject-specific didactic empiricism and theory is useful [49].

• What does this mean in relation to argumentative pupil texts?

In this diagnostic step, the extent to which the text fulfills or does not sufficiently fulfill certain criteria is examined. This process can raise further diagnostic questions about the pupils’ understanding of certain criteria. For example, the pupils’ argumentation can be used to determine what understanding they have of the concept of evidence as a technical basis and justification for their claim. When analyzing argumentative pupil texts with regard to their underlying competencies, reference can be made to criteria grids [10]. These are divided into the main categories of ‘language’, ‘argumentation’ and ‘structure’ and have already been explained in more detail in Section 2. The criteria grids include criteria abbreviations (e.g., R = spelling, MB = material reference), explanations of the individual sub-criteria and text examples that enable assignment and provide orientation.

3.

Giving feedback

The third diagnostic step is subsequently added to the model by Aufschnaiter, Münster and Beretz [49]. In this diagnostic step, the teacher asks themselves what feedback they will give the pupils about their performance. In addition, this is usually the first time that interaction takes place between the teacher and pupils.

There are many different terms for text feedback to pupils. In her study, Sturm [44] speaks of simple “comments” and Henke [54], for example, of “pedagogical final comments”, which teachers also use in the course of grading pupils’ texts to give hints on learning. In a meta-analysis, Rijlaarsdam et al. [55] emphasized that the strength of the effect of feedback on text quality is influenced in particular by the quality of the assessment criteria used.

Henke (ibid.) [54] explains that feedback is considered an individual support measure and that the teacher should at best pick out “two particularly conspicuous types of error” (ibid., 87) and use a direct approach to formulate advice. Positive aspects should also be mentioned in order to motivate pupils and give them the feeling that they can influence their own performance. However, the feedback should be authentic enough so that pupils do not gloss over deficient performance and do not make it appear contradictory in the event of a lower grade. Henke [54] also advises: “If, for example, a final comment threatens to degenerate into empty phrases, i.e., meaningless, formulaic expressions, it is better to do without it” (p. 89).

Written teacher comments supplement the marginal comments and text-immanent corrections in the pupil’s text and form a summarized assessment of the pupil’s performance [56]. In an analysis of teacher comments at the primary level, the authors found that these are often divided into a speech to the pupil including a positive appraisal, followed by an assessment of the writing performance against the background of selected criteria and finally a positive statement as encouragement. Only the form of the feedback was examined and not its effectiveness on writing success.

• What does this mean in relation to argumentative pupil texts?

Feedback to pupils on their argumentative text products includes linguistic textual corrections (a positive correction is recommended for weaker pupils), meaningful marginal comments on argumentation and structure and a final comment that provides direct feedback with examples anchored in the text. The previous results of the analyses should be consulted for this purpose.

4.

Clarifying causes

In the subsequent clarification of possible causes, an attempt is made to find explanations for the pupils’ performance based on the examination of performance and competency. This enables classification and provides indications of existing behavior [49]. Explanations for pupils’ behavior and deficits must be found in order to apply appropriate support measures and focus intervention. The identification of causes is, therefore, relevant in order to find the origin of the pupils’ existing competencies and to provide support where the source of deficient skills lies. Causes can be manifold and attributable to different aspects: for example, they can be rooted in the individual learning process, in the teacher’s lessons or outside of school—which is usually not visible to teachers. As a teacher, it can, therefore, make sense to enter into a dialog with other influential people: Pupils, parents, colleagues, etc.

Beretz [49] divides this into different types of causes: Situation-specific causes (e.g., careless mistakes, lack of understanding of the task), subject-specific causes (e.g., the complexity of the task or subject matter) and learner-specific causes (e.g., pupil perceptions, pupil resources, political attitudes). The differentiation and classification of these types seem useful in order to examine the extent to which some deficits are situation-dependent or were not communicated clearly enough in class or whether there are actual fundamental individual deficits among the learners. For this reason, relevant diagnostic assessments with appropriate support measures should ideally be made on the basis of several texts. This step also serves to evaluate the teacher’s own teaching [57].

• What does this mean in relation to argumentative pupil texts?

With regard to the diagnostic of written argumentation skills, the text assessment and deficit analysis are used as a basis for clarifying the causes. To this end, error patterns are used to try to find the causes for the lack of fulfillment of the assessment criteria. For example, if there is a lack of justification for an assertion in the text or the evidence is not valid, relevant or suitable, conclusions could be drawn about a lack of technical understanding. Nevertheless, it makes sense to look beyond the text product and place it in context in order to establish the cause: Some background knowledge regarding the pupil and reflection on the lesson can contribute to this.

5.

Drawing consequences

Finally, consequences are derived on the basis of the performance description, skills analysis and cause clarification. It summarizes which argumentative pupils’ skills still need to be further developed and how this could be promoted [49]. However, the further elaboration of the corresponding support measures is not part of the diagnostic process.

As in step four, consequences can also be drawn together in dialog with the learners: This can happen participatively with the pupils, the class, the parents, the teaching staff, etc. Together with the class, they can consider what content needs to be deepened and what needs the pupils have. These questions can also lead to pupils feeling more involved in the lesson, increasing their motivation and encouraging them to reflect on their own learning process. Teachers can draw consequences not only for pupils but also for their own lessons or beyond (e.g., method day, interdisciplinary cooperation with subjects, project week).

• What does this mean in relation to argumentative pupil texts?

The identification of deficits inherent in the text and the clarification of possible causes offer a good starting point for diagnosing pupils’ skills in argumentation and deriving support measures. At best, these are based on the text deficits, so that certain criteria in the areas of language, argumentation and structure can be addressed through individual support methods, renewed thematization and suitable tasks. In this case, it is worthwhile discussing the observations with the teaching staff and not only drawing conclusions for your own lessons or the respective pupils, but also addressing the criteria of successful written argumentation in German lessons, for example.

In school practice, at best, support measures are connected and implemented following the derivation of consequences and new writing tasks are set in the further course of teaching practice. At this point, the performance is recorded again, and the diagnostic cycle begins anew.

The model is an ideal-typical representation. The scope of individual diagnostic components varies depending on the product and pupil. It is relevant that the components of the diagnostic cycle do not have to be a linear sequence. Nevertheless, the text review and neutral description of pupils’ skills should form the starting point for diagnostic processes. At the end of the diagnostic cycle, work can continue with the results and the cycle can be restarted.

3. Current State of Research on Diagnostics and Feedback

For years, didactic research has been discussing the professional competence of teachers in diagnosing and promoting pupils’ skills against the background of the increasing heterogeneity of the pupil body and the growing need for support [58]. There are also numerous international studies that highlight the effectiveness of teacher feedback on pupil learning.

For example, Hattie’s [59] findings show that teachers have the greatest impact on pupil performance. Further research shows that the personality of the teacher has a greater influence than the type of school [60].

Meta-analyses by Graham et al. [61] showed that feedback from teachers has a significant effect on pupils’ writing performance. A study by Parr and Timperley [62] found a moderate correlation between pupils’ performance (field trip reports) and the quality of the teachers’ comments. The quality of the text evaluation of teachers with five years of professional experience does not differ from that of pre-service teachers [63,64]. Only when correcting different types of tasks was the quality of the assessment of experienced teachers found to be more consistent [64].

The current state of research shows many different correction habits and assessment criteria among teachers: there are no general standards for text assessment, and teachers assess very differently in terms of criteria and proceed differently [10]. Study results show that the assessment of pupils’ products for diagnostic purposes is less important for geography teachers than for performance assessment [37,65]. It is also controversial to what extent performance diagnostics through grading is an objective, reliable and valid measure of pupil performance [66,67,68]. These findings are in line with a study of geography teachers, which found that they do not conduct diagnostic support and do not use validated tools to diagnose pupil competence [37]. Authors such as Lambert and Balderstone [69] and Swift [70] have also investigated the use of methods to diagnose pupil performance in the UK geography classroom and argue for a combination of formative frontal assessment and summative assessment.

Findings from language education show that feedback is usually given by teachers and is mostly related to the text products and less to the process [44]. In a study on text evaluation and feedback with pre-service primary teachers at the end of their studies (total response rate: 51 teachers) and at the end of their first year in the profession, Sturm [63] also found that the majority of them are not able to adequately evaluate pupil texts without a criteria grid. In her study, she distinguishes between ‘text assessment’ and ‘comments’ (feedback) and finds that these are linked only to a limited extent and that the formulation of comments tends to emphasize positive aspects, regardless of how good the pupil’s text actually is.

In her study, she distinguishes between local and global aspects of the pupils’ texts. The local aspects of the text tend to include linguistic aspects and individual formulations, while the global aspects tend to include coherence, structural and content-related aspects as well as consideration of the basic idea of the text. In their study, teachers considered linguistic, structural and content criteria in their assessment. The extent to which criteria were taken into account can be attributed to the feedback style of the teacher [63,64].

She has also found that text feedback is often formulated in general principles and, above all, in a directive way, which makes it less pupil-friendly [63]. MacArthur [71] found that feedback formulated in dialogue had a greater effect on pupils’ text revision than direct formulation. Myhill, Jones and Watson [72] added that teachers tend to formulate only general principles or suggestions for improvement in feedback, rather than using specific examples from pupils’ texts to clarify deficits. Cho and MacArthur [73] also found that teacher feedback was often ambiguous, sometimes contradictory and not text-specific. In addition, recent studies have shown that geography teachers often provide ineffective feedback that lacks clarity, specificity and continuity [74].

Dempsey, Pytlik-Zillig and Bruning [75] summarize that pre-service teachers do not receive sufficient training in assessment during their studies. Furthermore, Henrichs [76] found that it is useful to promote diagnostic skills in higher education [76] thus laying the foundation for research in this area.

Teachers’ and pre-service teachers’ competencies and approaches to diagnosis and feedback on pupils’ argumentative writing have not yet been explored. In science education, only a few studies have shown that peer feedback has a positive effect on the improvement of argumentative texts [77,78,79,80].

Furthermore, there are no diagnostic, standardized tests for the teaching of geography. In the following, therefore, assessment criteria for the evaluation of argumentative pupil texts from different subjects are derived and combined with a focus on the didactic background of geography, in order to subsequently investigate the diagnostic behavior of pre-service geography teachers.

4. Methods

The empirical study is presented below by explaining the research questions and describing the methodology and sampling.

4.1. Research Design

The aim of this study is to investigate the diagnostic and feedback behavior of student teachers of geography regarding argumentative pupil texts. For this purpose, three research questions were developed, which were collected through a pretest, a text evaluation and open tasks for evaluation and evaluated qualitatively and quantitatively. The study is a qualitative exploratory study in a constructed, unnatural setting. The three research questions are presented separately below, followed by an explanation of the data collection methods and analysis criteria.

4.2. Research Questions and Methods

The first research question is “What experiences and attitudes do pre-service geography teachers have towards diagnosing pupils’ argumentative competencies in geography education??”. The respondents were asked to complete a pretest to find out about their experiences in the areas of argumentation, correction and diagnostics. Open questions on the understanding of the concept of correction and diagnosis, on criteria for a successful argumentative text, on the assessment of their own correction and diagnostic skills and on the relevance of the topic for geography education were answered in writing by the respondents. The texts were then analyzed using qualitative content analysis according to Mayring [81].

As a basis for answering the second research question: “How do pre-service geography teachers practice diagnosing pupils’ argumentative competence based on their argumentative texts?”, the test subjects were given an authentic text by an eighth-grade pupil at a secondary school (The pupil is 14 years old and in the eighth grade of a secondary school equivalent to a high school or grammar school), which they were first asked to correct and evaluate. This was performed digitally within a Word document, which enabled text-immanent corrections and marginal comments using the “Track changes” and “Comment” functions. They were given minimal contextual information. They were informed that the text was written by an eighth-grade pupil and that it took him 37 min to write the text. The pupil’s task was to write a letter to the mayor about a local conflict over the use of space by embedding various actors’ positions and expressing their own opinions based on this.

Regarding the selection of the text, it can be said that the pupil wrote it in a non-judgmental and voluntary scenario, the researchers have no direct connection with the pupil and the text was sent to the researchers anonymously. From a selection of several pupil texts, this one was chosen because it was one of the only ones that was comprehensible, readable and of a reasonable length. Another reason for the selection was that the text had a balance of strengths and weaknesses in each area (language, structure, argumentation) of an argumentative text.

Following the correction, the pre-service teachers were asked to evaluate the pupils’ skills in writing: “Now evaluate the pupils’ skills regarding argumentative writing: What is the pupil already good at? What is the pupil not yet so good at?”. Both the correction process and the writing of the evaluation were timed, with a maximum correction time of 15 min. The diagnostic assessments were evaluated using qualitative content analysis [81] and the diagnostic cycle model (Figure 2). The diagnostic that the pre-service teachers carried out was a planned status diagnostic, which measures the current state of pupils’ competencies on the basis of a text product. This was performed in a planned-for-interaction assessment [82,83], which corresponds to a planned diagnostic without systematic criteria and only uses a diagnostic question.

The test subjects were then asked to formulate feedback for the pupil so that the third research question “How do pre-service geography teachers give feedback on pupils’ argumentative texts?” could be answered.

In order to analyze the diagnostic judgments and the written feedback to the pupil of the individual pre-service geography teachers, the model (Figure 2) is used to examine which steps of the diagnostic cycle were taken into account and to what extent.

Table 1. Survey grid for the evaluation of diagnostic evaluations (source: own presentation).

Model Criteria	Guiding Questions for Evaluation
1 Recording performance	The pre-service teacher has… [ ] text-immanent corrections [ ] side comments [ ] Other which shows that they have reviewed the text product and recorded the pupils’ performance [10,49].
	These were formulated [ ] comprehensibly [ ] incomprehensibly [10,49].
2 Analyzing & interpreting competencies	In the diagnostic judgment, the pre-service teacher has … [10,49]. [ ] used criteria of good reasoning to identify positive aspects. [ ] used criteria of good reasoning to identify deficient aspects. [ ] identified competencies. [ ] used text examples to support their evaluation. [ ] listed a descriptive and an evaluative perspective [34].
	In the diagnostic evaluation, the pre-service teacher has… ____ linguistic criteria. ____ argumentative criteria. ____ structural criteria [10].
3 Giving feedback	In the feedback, the pre-service teacher… [ ] used a direct address [54]. [ ] used empty phrases 1 [54,63,72]. [ ] made textual references [54]. [ ] formulated dialogically [ ] directive 2 [63,71,84]. [ ] focused positive [ ] negative aspects [54]. [ ] focused local [ ] global text level 3 [63]. [ ] described a summarized evaluation of pupil performance [56].
	The pre-service teacher has named types of errors/criteria in the feedback _____ (number).
4 Clarifying causes	In the diagnostic judgment, the pre-service teacher has … [49] [ ] derived possible causes for the pupils’ abilities. [ ] derived no causes.
	The derived causes are … [49] [ ] situation-specific. [ ] learner-specific. [ ] subject-specific.
	The derived causes are… [ ] plausible against the background of the skills analysis [49]. [ ] implausible against the background of the skills analysis.
5 Drawing consequences	The pre-service teacher has derived from the diagnostic evaluation… [ ] no consequences. [ ] consequences for the individual support of the pupil [49]. [ ] consequences for teaching [49,57].
	The derived consequences for support are… [ ] suitable for the deficits & causes [49]. [ ] unsuitable for the deficits & causes.

¹ “meaningless, formulaic phrases” [54] (p. 89). ² Directive: instruction & instruction “you must…” (often affecting the local text level); dialogic: interrogative & subjective = prompted by questions & formulated subjectively “I think that you…” (often affecting the global text level) [63]. ³ The local text level refers to the word/sentence level (spelling, grammar, vocabulary, scope) and the global text level refers to the text level (overall idea, coherence, structure, argumentation).

illustrates the criteria used to evaluate the text correction, the diagnostic judgment and the feedback to the pupil and is based on the five steps of the diagnostic cycle.

To investigate whether the test subjects have taken into account the first step of the diagnostic cycle “recording performance”, the text evaluation is consulted and checked to see whether they have used text-immanent corrections, side comments or other comments (e.g., final comments under the text) in an understandable/not understandable way in order to generate a basis for analyzing and interpreting pupil competencies in step two. There, the diagnostic evaluation is examined using qualitative content analysis [81] to determine whether the pre-service teachers used criteria of good argumentation [10,81] to evaluate positive and deficient aspects in the pupil’s text, derived competencies of the pupil based on this, formulated a descriptive and evaluative perspective and used text examples for this. In addition, the number of assessment criteria of a linguistic, argumentative and structural nature named by the participants is counted.

For the third diagnostic step “Giving feedback”, the pre-service geography teachers’ feedback is analyzed separately. For this purpose, the criteria of formulation and the number of deficits named by the participants in the pupil text are examined for evaluation based on the research by Sturm [63]. Furthermore, the diagnostic evaluation examines the extent to which the test subjects deduce possible causes for the pupil’s abilities and explain these plausibly against the background of the skills analysis. Finally, the diagnostic evaluation is analyzed to determine whether the pre-service teachers formulate consequences, for example in the form of individual support or for their own teaching, against the background of the competence analysis and the reasons for the causes.

In the evaluation of the diagnostic and feedback evaluations, the criteria considered were simply counted and not evaluated as to how often or in what detail the individual criteria were described. In other words, if structural criteria were referred to three times and lengthy explanations of the deficit were given, the criteria were still scored once and not three times. This was not carried out, as the overall impression would not have changed and the research questions focused on design rather than quality.

The structural design of the feedback was also examined. Each participant’s feedback contains a different number of content-related statements. These are categorized into positive, neutral and negative statements about the text or pupil skills, instructions about the text or pupil skills, as well as the formulation of a learning objective or tips about the text or pupil skills. Based on this, feedback styles are typified. The extent to which text evaluation is related to diagnostics and feedback is then examined. This is important to examine in order to check the accuracy and scientific approach to the diagnostic on the one hand and to identify contradictory and disproportionate feedback on the other based on [54].

This is important to examine in order to check the accuracy of the fit and scientific approach to the diagnostic, on the one hand, and to identify contradictory and disproportionate feedback on the other based on [54].

4.3. Sample

This is a convenience sample, which means that the sample was selected neither purposively nor randomly but solely on the basis of accessibility, practicality and voluntariness. Therefore, there were no specific exclusion criteria for selection. The test participants were 20 pre-service geography teachers with an average age of 23.5 years. Each participant studied geography at a lower secondary level and had already studied the subject for an average of 4.5 semesters. The study took place in a bachelor’s programme at the University of Cologne (Germany) on planning geography lessons. Eight pre-service teachers are studying economics and politics as a second subject, five are studying German, five are studying biology and two are studying history. The study was anonymous, voluntary and non-evaluative. No content relevant to the study was taught beforehand in the seminar, and the pre-test (Section 5.1) was used to assess the participants’ previous knowledge and experience.

5. Results

5.1. Experiences and Attitudes of Pre-Service Geography Teachers

In order to answer the first research question, the pre-service geography teachers were asked in a pretest about their attitudes and experiences in the areas of correction, criteria for good argumentation, diagnostics and feedback. Figure 3 provides an overview of the pre-service geography teachers’ experiences.

Figure 3 shows that most of the test participants had already corrected a pupil’s text (but not in argumentative form) but did not feel confident doing it. Most pre-service teachers stated that they knew some of the criteria for correcting an argumentative text. The majority had already evaluated a pupil’s performance once, but not in all cases was it on a written product. Around half also have basic knowledge of evaluating pupils’ written work and over two-thirds of pre-service teachers have already given pupils feedback on their performance—half of them also on written work. Finally, the evaluation shows that none of the test participants had any previous experience with the diagnostic of written pupil performance.

In addition, the pre-service geography teachers were asked an open question regarding their understanding of the term “diagnostic”. In relation to the model of the diagnostic cycle (Figure 2), six pre-service teachers did not name any aspect and seven each named the recording of performance and the analysis and interpretation of competencies. Only two participants each mentioned feedback and justification of causes as part of diagnostics and three mentioned consequences in the context of diagnostics. Almost all pre-service teachers identified diagnostics as an action and an assessment process. Most explained that diagnostics is an examination of pupils’ strengths and weaknesses and thus pupils’ skills. Some reported that only the weaknesses were highlighted in the diagnostic process. Four pre-service teachers explained that “diagnostic” is an evaluation or determination of a fact. Three test participants stated that a diagnosis is based on previously obtained data and facts—only one test participant stated that diagnostic criteria were used. Many definitions are also formulated imprecisely or too generally: “When you examine and evaluate something” (respondent 1).

If the theory on the term “diagnostic” [34,35,36,38,41] is used and compared with the answers of the prospective teachers, it is noticeable that there are also some misconceptions and one-sided ideas about the term. For example, one test participant confused the term with feedback: “I understand diagnostics to mean feedback from the teacher. The diagnostic helps the pupils to expand their competencies and progress thematically” (respondent 2) or another test participant with the term correction: “Checking the pupils’ tasks and analyzing them, which can improve pupil development and teaching” (respondent 3). The latter statement also shows that there is an understanding of the didactic purpose of diagnostics. However, it is not mentioned that the diagnostic process must be followed up with feedback so that pupils receive feedback on their competencies.

The participants were also asked to what extent they find the diagnostic methods in geography education important—16 out of 20 pre-service teachers found them relevant. The most common reason given for this was that diagnostics can provide support, and pupils receive feedback on their performance. The competency of argumentative writing was emphasized several times as being particularly relevant for geography lessons: one test participant described this as “Very important, as pupils should be able to express themselves argumentatively. The ability to act is particularly important here, as pupils should learn to argue and then find problem-oriented solutions” (respondent 4). It was also emphasized that diagnostics can help to examine pupils’ understanding and ideas regarding a subject matter in order to subsequently adapt their own teaching accordingly. Only two people felt that diagnostic methods in geography education for written argumentation skills were irrelevant/partially relevant, but without giving a reason. Two others emphasized that they only found the activity relevant for tasks in requirement area 3—which includes argumentative writing—and another person would make the relevance dependent on the level and type of school (without elaborating on this).

In order to examine whether the diagnostic of pupils’ written argumentation skills by pre-service geography teachers is related to their ideas and subject-specific knowledge, an open question was used to investigate which assessment criteria for an argumentative text the test participants named. Each test participant named an average of 2.2 criteria that they considered relevant for a successful argumentative text. Referring back to the criteria grid [10], which divides the three broad areas of language, argumentation and structure into sub-criteria, it was found that most of the participants’ mentions were in the area of argumentation. Here, the most frequent mentions were that pupil positioning (7), complexity (5) and the presentation of different positions of the actors played a major role. Most pre-service teachers mentioned that “both sides (i.e., pros and cons) must appear in the text” (respondent 5). It also became clear that the future teachers could only roughly describe their criteria and did not use any specialist vocabulary: Terms such as suitability were paraphrased as “comprehensible and conclusive” (respondent 6) or validity with the use of “correct data” (respondent 7). In two cases, “content” was mentioned as a criterion and no further differentiation was made. Half of the test participants described the structure/composition of the text as an important criterion. There were only a few mentions in the area of linguistic criteria: Here, spelling and expression were mentioned twice each and grammar and text procedures (“good sentence beginnings”) once each. Overall, the picture that emerges is that the pre-service teachers focus on the technical content and that the expectations of an argumentative text primarily include the presentation of different positions, their own positioning and the appropriate arrangement of arguments.

The test participants were asked to provide a written assessment of their own correction and diagnostic skills regarding argumentative pupil texts. As the evaluation with the Likert scale already showed, only a few participants had already gained experience in these areas and would classify their correction skills as “rather weak” to “average”. Only two test pre-service teachers described their skills as “good”. Overall, the test participants’ statements resonate with the idea that practical experience plays a decisive role. For example, one test participant described: “I have had rather little experience in this regard and although I am confident that I have a certain basic structure, I would say that I can only gain the necessary finesse through longer practice” (respondent 8). Two participants also emphasized that the correcting teacher should be objective and open to the results and opinions of pupils that differ from their own. The majority of pre-service teachers rated their own correction and diagnostic skills as “poor” or “unsatisfactory”.

5.2. Conception of the Diagnostic Evaluation

The second research question: “ How do pre-service geography teachers practice diagnosing pupils’ argumentative competence based on their argumentative texts?” was used to investigate the extent to which the pre-service teachers intuitively considered the steps of the diagnostic cycle in their diagnostic evaluation.

The test participants took an average of 05:11 min to write their diagnostic evaluation, ranging from 02:58 to 07:28 min. The texts comprised an average of 81.55 words in a range between 22 and 158 words. The diagnostic evaluations were partly written in text form and in a few cases in bullet point form.

In the following, the results are assigned to the diagnostic cycle and its individual steps, starting with “Step 1: Recording performance” (see Table 1).

5.2.1. Recording Performance

The text evaluations and corrections were examined in order to check the extent to which the pre-service geography teachers had recorded the pupil’s performance.

Figure 4 shows the errors corrected by the test participants in the pupil text, assigned to the main categories of the theoretical criteria grid according to Steingrübl and Budke [10]. In total, 19 out of 20 test participants evaluated the pupil text in the form of text-immanent corrections or side comments. Of these, eight pre-service teachers wrote text-immanent corrections and 15 wrote side comments. Five participants also wrote additional final comments or merely underlined errors.

It is clear from the diagram that 78% of the 294 corrections and text evaluations made included linguistic aspects. Of these, the most frequently corrected aspects were expression (73), spelling (59) and grammar (44). There was only one mention of deficient use of tense and three mentions regarding poor text procedures. From a subject-specific perspective, it should be mentioned that there was no evaluation of technical terms. A total of 18 out of 20 pre-service teachers evaluated 35 linguistic aspects. In total, 13 participants made evaluations in the argumentative area: justification (13) and pupil positioning (6) were most frequently criticized. Criteria such as validity, relevance, suitability, positions of the actors involved in the conflict and spatial and temporal conditions were evaluated sporadically and the material reference in the text and the presentation of the controversy were not evaluated at all. The validity of the evidence and the suitability of the validity relationship are often summarized in everyday language under “justification”. However, this should be separated from a scientific point of view as the lack of complexity was once criticized in the side comment. In the main category “structure”, 13 pre-service geography teachers also evaluated criteria: with 15 mentions, deficits such as the structuring and order of the individual arguments in the main part were most frequently noted; deficits in the text type and the conclusion were found eight times each. In order for these criteria to be internalized and evaluated separately, the pre-service teachers need a foundation in argumentation theory.

In the course of the analysis, it was not checked to what extent the text-immanent corrections, side comments, etc. correspond to the correctness or are understandable. The aim here is merely to provide a rough overview of the focus of the participants in the diagnostic step “Recording performance”.

5.2.2. Analyzing and Interpreting Competencies, Clarifying Causes and Formulating Consequences

To check whether the “Analyzing and interpreting competencies” diagnostic step was taken into account, the written diagnostic evaluation was evaluated for each test participant. Figure 5 provides an overview of the evaluation of the future teachers’ diagnostic evaluations in the diagnostic steps “Analyzing and interpreting competencies”, “Clarifying causes” and “Drawing consequences”.

It is evident that the majority of pre-service geography teachers used criteria for evaluating argumentation to identify positive aspects and deficient aspects of the text. As a result, all participants used an evaluative formulation regarding the text and the positive and improvable aspects present, but only seven people also used a descriptive formulation. An example of this was as follows: “The pupil argues with arguments for and against. He alternates between them and presents one argument for and one argument against” (respondent 7). In addition, half of them inferred competencies of the pupil and only three pre-service teachers used text examples as explanations in their diagnostic evaluation.

The examination of the pre-service teachers’ diagnostic texts with regard to the assessment criteria used for a successful argumentative text is illustrated in Figure 6.

A total of 112 criteria, an average of 5.6 criteria per person, were used to evaluate the text and pupil competencies. A total of 15 pre-service geography teachers used 36 criteria to emphasize positive aspects of the pupil text. Of these, complexity and pupil positioning were used most frequently. All test subjects used more than twice as many evaluation criteria (n = 76) to describe deficits in the pupil text. It is striking that 14 people criticized the structure and frequently the pupil positioning and expression. The item “Other” is assigned in the graphic partly to the linguistic and partly to the argumentative main category and under this, the test participants named criteria such as the number, persuasiveness and invalidation of counterarguments, the synthesis of arguments, repetition of content and the pupil’s reception competence. It can be stated that technical terms, spatial and temporal conditions under which the arguments apply, and the validity of the evidence did not play a role in the pre-service teachers’ evaluation of the text.

In addition to the presentation of the individual criteria, Figure 7 shows the relationship between the criteria evaluated by the pre-service geography teachers within the main categories of language, argumentation and structure.

Overall, almost half of all the evaluation criteria were of an argumentative nature and around a quarter each were of a structural and linguistic nature. If the ratio of negatively and positively evaluated criteria is considered, it is noticeable that this is relatively balanced in the area of argumentation and diverges in the language and structure evaluation.

When analyzing the statements of the test participants, it was often said that the pupil “picked out good arguments from both sides”, which illustrates a misconception: there is sometimes the understanding that the argumentation consists of two sides (pro and con), ignoring the fact that the focus is on the different opinions of the actors in order to create a multi-perspective argumentation and not merely on the presentation of pro and contra arguments.

If competencies were derived on the basis of the evaluation criteria, they were mostly with sentence fragments such as: “the pupil still has difficulties/problems…”, “the pupil is able to…” or “the pupil still has to learn…”. Only one statement was specifically linked to the concept of competence: “It is clear that the pupil has good assessment skills” (respondent 3), yet this was neither explained nor elaborated on. A lack of competencies in structuring the argumentation was mentioned most frequently: “It is difficult for him to express his thoughts in a structured way in text form, which is why his text sometimes seems arbitrary and unstructured” (respondent 4) or the linguistic implementation was criticized: “In general, the pupil has some problems with formulations and spelling” (respondent 9). This result matches those of the text evaluation or text correction (Figure 4). Only one test participant tried to find causes for the deficits:

“If you had more information, you could also go into whether he basically let his opinion flow in, so that he doesn’t even mention the positive aspects because he thinks the idea itself is “stupid”” (respondent 10).

In addition to the text, it may also be relevant to collect further background information about the pupil, their opinion and attitude or the subject matter in order to find out the causes of their competencies. Furthermore, reference is made to the learning or subject-specific causes that relate to the pupil’s attitude regarding the subject matter.

No consequences were drawn for their own teaching and only a few test participants found approaches to formulate consequences for the pupil. These consequences were formulated, for example, in the form of learning objectives: “He must learn that you write “that” when you can replace it with that or which. He must learn that literal speech should be noted with (respondent 11) and in this diagnostic evaluation it was added, “It would be helpful if Peter read a lot in his free time to improve his competency”. It is clear that the impression prevails here that reading would improve written language competency and is, therefore, evaluated as a “consequence for the pupil”. In another diagnostic evaluation, it was pointed out that the pupil should look again at the rules for argumentation and another person formulated as an unspecific consequence: “The pupil has a good concept that can be built on well with the necessary solutions” (respondent 8).

Six test participants already formulated tips and suggestions for improvement in their diagnostic evaluation on how the pupil could have structured his argumentation better, for example: “He could have used a “ping pong” or “hourglass” model as a guide, so that he would work towards his own position and it would be clear to the reader which opinion the author represents” (respondent 6). All suggestions for improvement related to structural aspects or wording. Furthermore, empty phrases such as “The pupil is able to convert his arguments into written form” (respondent 11) were sometimes formulated in the diagnostic evaluations, which had no function in the evaluation and diagnostic.

5.3. Design of the Feedback

In the course of the third research question “How do pre-service geography teachers give feedback on pupils’ argumentative texts?”, the future geography teachers formulated personal feedback to the pupil on their text product. This was evaluated using various criteria and is presented below. On average, the feedback for the pupil comprises 84.3 words, with the shortest being 16 and the longest 176 words.

Figure 8 shows that the pre-service geography teachers gave more praise (positive aspects of the text and pupil competence) than criticism (negative aspects of the text and pupil competence) in their feedback. Many participants gave advice on how the pupil could improve the text next time (text tip) or how they could further develop their competencies (competency tip). In addition, concrete learning objectives were formulated eight times and instructions were given 29 times on how the text should be designed (instruction text) and which competency the pupil should promote (instruction competency).

Figure 9 illustrates how the pre-service teachers structured their feedback evaluations. For the evaluation, the individual feedback texts were divided into individual statements and assigned to categories, each of which contains an aspect of the feedback to the pupil. The x-axis shows the course and structure of the feedback evaluations. Each column refers to a statement made by the participant (e.g., the first column shows the statements with which the pre-service geography teacher began their feedback). As the participants used different numbers of statements, the number of these towards the “end” decreased. For example, one test participant only used three statements, and the longest feedback included eleven statements. On average, the pre-service teachers used 6.5 statements in their feedback texts. The y-axis of the diagram shows the categories to which the individual statements are assigned.

The diagram (Figure 9) shows that most of all pre-service teachers begin their feedback by mentioning a positive aspect. After that, there are no clear characteristics, which may indicate different feedback styles. In terms of feedback styles, a distinction was made between predominantly positive/negative statements, a predominant focus on the text/competency and the formulation of advice or instructions. To this end, feedback styles could be determined inductively, the characteristics of which are shown in

Table 2. Feedback styles (Source: own illustration).

Feedback Styles	Description	Positive vs. Negative Focus	Text vs. Competency Orientation	Advice vs. Instructions
Advising	Makes concrete suggestions for improving the text and competencies in a dialogical manner. The wording and focus are positive. Praise is often encouraging at the beginning and again at the end of the feedback.	positive	both	advice
Criticizing	Emphasizes mainly negative aspects and criticizes. This can be related to the text and competencies. No advice is formulated, but instructions are given if at all.	negative	both	(instructions)
Instructing	Formulates direct instructions to be followed up and emphasizes negative aspects for his explanation. This can be related to the text and the competencies.	both	both	instructions
Balancing	Balances positive and negative aspects at text level. These are presented alternately and there is hardly any tendency to summarize the assessment. An attempt is made to evaluate performance holistically and objectively.	both	text	both/nothing
Supporting	Focuses on both negative and positive aspects, particularly in relation to competencies. Learning objectives are linked and support options are considered.	both	competency	Rather advice

.

The feedback from seven pre-service teachers could be assigned to the advisory feedback style—this fits with the results of the previous Figure 8, where it is clear that predominantly positive aspects were reported back. The feedback from five participants corresponded most closely to a balancing style, as it balanced praise and criticism. Four people used the directive feedback style by giving mainly direct instructions and two test participants each formulated their feedback in a predominantly critical or supportive manner.

Of 18 pre-service teachers, almost all of them chose to address the pupil directly in their feedback evaluation. A total of 14 people and thus almost ¾ formulated their feedback in a clearly directive manner (instructing, demanding), four in a clearly dialogic manner (questioning, subjectively formulated) and two neutrally. Only one person made a concrete reference to the pupil text in their feedback by praising specific positive examples of the integration of topics and giving structural advice for improvements:

“You’re representing the citizens’ side, if I‘ve got that right. Therefore, next time I would mention the citizens’ arguments at the end, because the many minor objections from the soccer club that you have mentioned almost make you lose sight of the arguments from the citizens’ side, although I think the arguments are strong and that you have picked them out well. Animals, nature, loss of local recreation are good arguments. If you had mentioned these after you had mentioned the footballers’ arguments, the FC’s arguments would have already been plausibly refuted” (respondent 12).

This quote can also be assigned to an advising feedback style, as the formulations (e.g., “therefore I would”) are formulated dialogically, praise is given and at the same time, advice is given instead of instructions. Within the feedback evaluations, it was found that 12 pre-service teachers used a predominantly global approach to the text, which relates in particular to the structure and argumentation. Only two people focused primarily on the local text level and thus on individual linguistic deficits. The six others used a mixture of both approaches. In addition, all but three pre-service geography teachers used their feedback to formulate a summarized overall assessment of the text. This was often introduced by text procedures such as “overall” and “summarizing” and described at the beginning or end of the evaluation. Approximately half of the pre-service teachers used phrases in their feedback that had no meaning or relevance in terms of content. Most frequently, the introductory sentence emphasized that the pupil had formulated “good arguments”. In addition, it was occasionally reported that “the text was thematically good” and was “read with great interest”. The pupil had also “found good arguments in the text”. In addition, some statements were also rated as empty phrases if they did not match the feedback, such as one person who criticized the expression, grammar and conclusion and also advised the pupil: “It is helpful if you start reading books in your free time” (respondent 11).

Figure 10 illustrates which criteria pre-service geography teachers used within their feedback statements to provide guidance regarding the pupil’s performance. On average, each participant used 4.75 criteria within their feedback. Overall, the number of positive and negative feedback criteria is balanced (46% positive, 54% negative). It is striking that half of all criteria reported back are assigned to the main category of argumentation. In contrast, only 19% of all criteria were reported in the linguistic main category. The test participants predominantly reported positive criteria in the area of argumentation and predominantly negative aspects in the other two categories. The most positive aspects were pupil positioning and the complexity of the argumentation. The most frequent points of criticism were in the area of structure. Here, it was often criticized that the pupil had already stated his own opinion in the main part, but in the pre-service teachers’ understanding, this only belonged at the end of the argumentation.

6. Discussion

Following the presentation of the results, these are discussed and interpreted below against the theoretical background and the research questions.

First, the experiences and attitudes of the pre-service geography teachers towards diagnostic methods in geography education were examined. In the study presented, it becomes clear that the pre-service teachers have had limited contact with text evaluation, diagnosis and feedback on written argumentative pupil texts and tend to rate their skills low. This overlaps with the results of a geography teacher study on text evaluation skills, where respondents stated that they usually only acquired these skills in their everyday work and not during their studies [10] and the findings of Dempsey, Pytlik-Zillig and Bruning [75], which emphasized the insufficient training in the evaluation of text products by the university.

The study then examined how the pre-service teachers carried out text evaluation, diagnosis and feedback on the argumentative pupil text. For this purpose, the model of the diagnostic cycle (see Figure 2) was developed and applied in the evaluation in order to check to what extent the participants take into account steps of the cycle and which criteria they use to evaluate the text. For this purpose, the diagnostic evaluation was examined on the one hand and, on the other, open-ended questions were used to ask about experiences and attitudes. In their definition of “diagnosis”, the pre-service geography teachers only partially name aspects of the diagnostic cycle (see Figure 2), with the most frequent being the recording of performance and analysis and interpretation of competencies, but hardly any feedback, justification of causes and derivation of consequences.

Nevertheless, only a few people derived specific pupil competencies from the text product, instead merely sticking to describing the errors found in the text. This may be due to an understanding that daily school life is often about performance evaluation, and this can take place without the cycle of steps three to five. This assumption can also be derived from the pre-service teachers’ understanding of diagnostics: They often see diagnostics as an assessment of a performance and less as a process that leads to support and adjustment of teaching. Deriving consequences from the correction and diagnosis was almost non-existent among the test participants, and if it did, it only concerned the individual support of the pupil and not possible consequences for the lesson. Thus, there is still a lack of understanding here that diagnostics also serve to evaluate one’s own teaching and that this is important for the long-term support of the pupil. This is clearly illustrated by a quote from some feedback: “It’s helpful if you start reading books in your free time” (respondent 11)—this shifts the responsibility for promoting skills from the teacher in the classroom to the pupil’s own responsibility in their free time. Although the survey took place as part of a didactic seminar on geography lesson planning, many test participants had not yet internalized the fact that geography education also serves to promote pupils’ written argumentation skills [65,85]—this should be taken into account in university teaching in the future.

In addition, the results show that the separation between the individual steps of the diagnostic cycle of correction, diagnosis and feedback is not clear to many pre-service geography teachers. The incomplete understanding of diagnostics leads to individual steps of the diagnostic cycle and important evaluation criteria being omitted or skipped, resulting in an imprecise assessment of competence and subsequent incomplete feedback.

Figure 11 compares the criteria used by the pre-service geography teachers for text evaluation, diagnostics and feedback in the main categories of language, argumentation and structure. It is clear that the number of criteria used in the text evaluation is significantly greater at the beginning of the survey and that more criteria were used to identify negative aspects than in the diagnostic and feedback. Especially in the feedback, argumentative aspects are emphasized more positively than they were described in the diagnostic evaluation.

In the text evaluation and performance evaluation, 78% of linguistic criteria were evaluated or corrected (see Figure 4 and Figure 9). This is in line with the results of Steingrübl and Budke [10] and could be due to the fact that the participants understand linguistic corrections regarding the term “correction” and that this is easier for them to correct than argumentative aspects. When it comes to linguistic correction, they may be able to draw on their own school experiences. In addition, Steingrübl and Budke [10] have already found that geography teachers have difficulties in evaluating and formulating deficits based on argumentative criteria in and on the text during correction. This is also evident among pre-service geography teachers. In the diagnostic and feedback evaluations, on the other hand, argumentative evaluation criteria were used more frequently, which may be due to the fact that formulation is more common there and pre-service teachers can paraphrase their comments in more detail. Linguistic aspects are rarely mentioned in the explanations. It is, therefore, subliminally assumed that the pupils know enough about this and that the text corrections provide sufficient feedback and do not need to be explained. Signal words such as “spelling” or “formulate better” are sufficient. It is clear from the findings that the test participants need to know both which criteria they use for evaluation and how they apply them for text evaluation and diagnostics and formulate them in a way that is appropriate for the pupil.

The pre-service geography teachers most frequently named pupil positioning, complexity and the presentation of different actor positions as criteria for successful argumentation. This is also reflected in the diagnostics and feedback. The analysis also showed that the participants used few criteria for evaluation and feedback overall—5.6 criteria per test participant and thus only ~25% of the 22 criteria derived from the literature for evaluating argumentative texts in geography education [10]. In addition, the test participants rarely used argumentation-theoretical terms in the diagnostic and feedback to formulate successes or deficits in the text or in pupil competence: For example, validity was described as “good content”, “good arguments” or “correct data”, which means that the pupil lacks a concrete description of what exactly is now good or in need of improvement about the argument. The lack of knowledge of argumentation theory concepts can also be a reason for the empty phrases used in the feedback evaluations. It can, therefore, be assumed that the pre-service geography teachers are only partially aware of the criteria of a successful argument. This can also be seen in the geography-specific criteria.

The pre-service geography teachers hardly use any subject-specific criteria for their feedback, but these are relevant for the evaluation of argumentation in geography education. This concerns subject-specific terminology [27,28,29,30], spatial and temporal conditions [2], material reference [6,32] and validity of the subject-specific evidence [5,28]. This is in line with the findings of Steingrübl and Budke [10], who found that geography teachers used little to no geography-specific criteria in their text evaluations. In order for teachers to be able to evaluate subject-specific content, they must first have penetrated it themselves and then be able to express it in a way that is appropriate for pupils.

The last research question was used to examine the design of the feedback to the pupil. The feedback texts from pre-service geography teachers to pupils are very diverse (style, complexity, criteria). Different feedback styles were identified. The present study and that of Sturm [62] showed that teachers tend to emphasize positive aspects when giving feedback, regardless of the quality of the text. One reason could be that the participants do not want to have a demotivating effect on the pupil. This is also shown by the use of empty phrases, which they may use to relativize criticism. Both the present study and that of Cho and MacArthur [73] show that text examples are rarely used in both diagnostics and feedback to show deficits by way of example. This is problematic, as pupils need these in order to understand the teacher’s comments.

The pre-service teachers seem to have a rough knowledge of feedback, as they use a positive start and a direct address to the pupil, the evaluation of competencies based on criteria and the formulation of instructions and advice. Jost et. al. [56] described this as a classic breakdown of feedback with the exception that they add a positive appreciation at the end, which not all of the test participants did. The majority of pre-service geography teachers formulated their feedback in a directive manner—according to Sturm [63] and Bouwer and van den Bergh [84]—and thus less pupil-friendly. Only a few participants remained on a local consideration of the text level and focused on specific linguistic deficits when giving feedback. This could be due to the fact that they assume that the pupil receives the corrected text evaluation at the same time as the feedback evaluation and that this is also used as feedback.

Overall, the results in the areas of text evaluation, diagnostic and feedback show how relevant the five-step diagnostic cycle (Figure 2) is, as different criteria are used by the pre-service geography teachers for evaluation in each sub-step. This means that the final feedback is complemented by the text evaluation and the diagnostic. The model of the diagnostic cycle (Figure 2) and the results also make it clear that in most cases the feedback is strongly based on the diagnostic and this in turn is based on the text evaluation, meaning that the three steps should always build on each other logically in the order text evaluation/correction—diagnostic—feedback.

The findings of the study confirm previous findings from other subjects and studies with teachers. Sturm [63] and Bouwer et. al. [64] have so far shown that the quality of text evaluation of teachers with five years of professional experience hardly differs from that of pre-service teachers. This suggests that the “learning-by-doing” [10] or “learning on the job” mentioned in teacher interviews on text evaluation is not sufficient. Nevertheless, it is also clear that many pre-service geography teachers find the support of written argumentation skills relevant for geography education and, like Schrader [35] and Weber [36], see them as a starting point for the support and optimization of learning processes. This is a good starting point for the support and thematization as well as motivation of future teachers to further their education in this area and to implement this in their geography lessons.

Methodological Criticism and Limitations of the Study

The study was able to use the research questions on the attitudes, diagnosis and feedback of pre-service geography teachers regarding an argumentative pupil text to find out how they organize diagnosis and feedback and what criteria they use to evaluate the text. As this is a qualitative study with a pupil text example, no general statements can be made about the quality of pre-service geography teachers’ diagnostic and feedback skills.

The theoretical models, such as the criteria grid and the diagnostic cycle, were not shown to the participants beforehand or given to them to help them complete the tasks. This could have altered the results. In this research project, the models only served as part of the evaluation methodology and as a preview of possible support measures for prospective geography teachers. The research design can be criticized in that there is no comparison between a group of subjects who are diagnosed with knowledge of the criteria grids and the diagnostic cycle and a group who are not. However, this comparison would have addressed a further research question that would have evaluated the models. The purpose of this basic experimental research was to present the current state of diagnosis and feedback design and to provide a starting point for quantitative, hypothesis-driven research into the use or non-use of the models in relation to behavioral practice.

The pre-service geography teachers received a text from an unknown pupil in an unnatural setting without explicit contextual information about the background of the pupil or the lesson, which in turn could be a reason why the area of “reasoning” was hardly present in the diagnostic evaluation. For this reason, however, this study did not evaluate the quality of the diagnostic, but merely took a first look at the procedure.

In text evaluation, the use of side comments and text-immanent corrections may also depend on the file format: There may be a different operation for a handwritten correction, this is also shown by results from a teacher interview study by Steingrübl and Budke [10]. It was also not mentioned that the pupil would receive the text evaluation in addition to the feedback—perhaps the text-immanent corrections and side comments would then have been more detailed. Methodologically, only the feedback evaluation was evaluated as feedback to the pupil in the evaluation of the results; however, this also includes the text evaluation. Nevertheless, it plays less of a role in how they correct but determines if they correct at all and thus prepare themselves for the diagnostic and feedback.

Furthermore, there were no criteria grids for evaluation. If this had been the case, step 2 (analyzing and interpreting competencies) might have been more intensive.

7. Conclusions

The importance of addressing and promoting argumentation in geography education cannot be doubted, as this is necessary to penetrate the subject of geography. In order to pursue this, teachers need to have knowledge of argumentation theory and be aware of the criteria in order to penetrate, communicate, assess, feedback and support pupils in the subject itself. In order to provide subject-specific support for learners’ argumentation skills in the classroom, their competencies should first be diagnosed. The theoretical introduction shows that the subject-specific diagnosis of competencies for the production of written argumentation in geography on the basis of argumentative pupil texts is very extensive and demanding.

The study shows that prospective geography teachers have little contact with text assessment, diagnosis and feedback during their studies and rate their competence in this area as low. Difficulties already exist in the definition of ‘diagnosis’, which they see as a performance assessment rather than a starting point for developing pupils’ competencies, which may explain why they only partially carried out the steps of the diagnostic cycle with the argumentative pupil text and did not consider relevant steps such as exploring causes and inferring consequences. Probably the most important observation of the study, however, was that pre-service geography teachers hardly used any geography-specific assessment criteria in any of the practices analyzed, which can be a major problem for pupils’ subject-specific learning. The analysis of the pre-service geography teachers’ feedback texts also showed that only a few criteria were reported back and that these were unspecific, without text examples and without reference to geographical content. Furthermore, it became clear that the feedback comments were very diverse and that different types of feedback could be identified, which suggests that this is not (yet) being addressed at the university.

It was not possible in the study to find out how the pre-service geography teachers would deal with other text examples. Furthermore, this is not a study of the participants’ competencies but only an initial exploration and insight into the practices of dealing with argumentative texts in geography class. As this was an experimental, unnatural setting with an unknown pupil, it would be interesting for future research to see how the pre-service teachers would behave in a natural setting and with support and theoretical instruction on the diagnostic cycle and assessment criteria.

Nevertheless, the following findings and recommendations can be derived from the study: (prospective) teachers must be enabled to use diagnostic tools and formulate appropriate feedback, as this correlates with learners’ success. Pre-service geography teachers must, therefore, be taught content-related competencies as well as competencies in the area of diagnostics and feedback in order to design and evaluate their own lessons in their later professional life.

This makes it all the more important to consider, develop, apply and evaluate criteria grids for assessing pupils’ argumentative texts in order to fully and specifically assess and report on this complex linguistic action in geography lessons. These save additional time and can indicate the expectations of a text by embedding it in the lesson. In addition, the use of the diagnostic cycle (see Figure 2) can help (future) geography teachers to better understand the scope and structure of a diagnosis, strengthen their lesson evaluation and self-reflection, and better support pupils. A new focus in diagnosis should aim to draw didactic consequences in order to provide targeted support in the classroom. The three steps of text assessment, diagnosis and feedback should be taken into account in order to derive effective support. In addition, (trainee) teachers need to be encouraged in their feedback judgements to embed more concrete text examples, to name argumentation-theoretical terms and to focus on geography-specific criteria such as technical terms, spatial and temporal conditions and the validity of evidence.

Given that space, spatial reference, different perspectives and complex reasoning are at the heart of geography education, a new focus is needed in teacher education. Diagnosis and feedback need to be increasingly practiced on a subject-specific basis. Knowledge of subject content, subject-specific argumentation criteria, argumentation theory and practice (how do I correct/assess/report aspects?) should be taught during the course of study. In addition, geography teachers need to be encouraged to know and use appropriate methods to promote argumentation skills: Digital learning units from the DiGeo (DiGeo is a joint project of the Universities of Duisburg-Essen, Frankfurt am Main and Cologne, funded by the German Ministry of Education, to develop skills in the use of digital geomedia in teacher training in the subject of geography. Within the project, digital learning units were designed to promote teachers’ argumentation skills and methods for using them in geography classes. To this end, theoretical principles of argumentation theory and geography didactics were combined and methods for promoting argumentation as well as for diagnosing argumentation skills were developed). collaborative project (https://digeo-oer.net/doku.php?id=start accessed on 10 August 2024) can support this.

Overall, further research is needed to fill gaps in both theoretical research and practical implementation in schools, especially in the area of geographical learning diagnostics.