Diagnostic and Feedback Behavior of German Pre-Service Teachers Regarding Argumentative Pupils’ Texts in Geography Education

Comments

Correction

When I started to read the manuscript, I was hoping that it would be an intriguing and welcomed piece of research namely to the field of geography education. Unfortunately, there are fundamental issues in the manuscript that need to be addressed and clarified. Firstly, much of the results does not explicitly have to do with geography education perse but is more about ‘language and grammar;’ that is, the geographical content of the results should be in focus instead, such as the use of geography-specific criteria (line 932). In other words, the results do not include geographical content, although it is stated that “diagnostics and feedback must increasingly be practiced on a subject-specific basis” (lines 1040–1041). For instance, it would have been interesting to know what sorts of actor positions the 8^th grader took in his text. Secondly, another major revision has to do with the fact that mainly German references are used, which does not take into account international audiences. Thirdly, there are many careless errors and misspells, which is why many minor revisions are to be executed in order to enhance the quality and logic of the manuscript. 

Thank you for your comments. I can understand the aspect that the linguistic component erroneously takes up a large proportion of the research. The research is embedded in the research area of language in geography lessons and is part of language-sensitive geography lessons, where the guiding principle is that language and geographical learning are interrelated and interdependent.

If it has not become clear, it was not that I analysed the student text, but the students. Thus, the focus on language is due to the students' diagnosis and is a result of the research that I cannot change. Accordingly, I cannot make any changes to the effect that I focus on the subject-specific aspects - I had to analyse the students' diagnostic judgement and the design of the feedback and since they did not set the focus here, I can only emphasise in the discussion that there should be a stronger focus on the geographical content and that this should already take place in the form of awareness-raising and support for students at the university.

I have added English literature and reduced the findings in the German-language section.

I have adapted the language of the manuscript.

What concerns me is the fact that most of the references are in German and not in English. When it comes to international readers, they cannot read and therefore use these German sources; that is, readers are not to benefit of the sources used. When aiming one’s research to international audiences, I think it is thus important to take advantage of prior research published in English—otherwise, you could publish your manuscript in a German journal. Therefore, I would like to suggest that you will go through your sources—which you have many, that is, altogether 87 sources of which about only 25 is in English—and check whether every German source is needed in the manuscript and try to add some English publications in order to have a better balance with the German and English sources.

I have added English literature and reduced the findings in the German-speaking area.

Nevertheless, I have referred in particular to the German literature and research, as the school curriculum and the demands placed on students by the German educational standards are certainly different from those in other countries and the international studies are therefore less comparable with my results.

In terms of the main title and its grammar, should it be “pupils’ texts”. In addition, instead of ‘university students,’ consider using the word ‘pre-service teachers,’ or even ‘German pre-service teachers’ because the research is about a case-study of twenty German pre-service teachers (e.g., “Diagnostic and feedback behavior of German pre-service teachers regarding pupils’ argumentative texts in geography education”). Although geography education is very loosely linked to the manuscript.

Many thanks for the hint! I have also made these adjustments and changed the title.

I would still like to emphasise that the focus of my dissertation, from which this manuscript arose, is on language acts in geography lessons, which is a recognised field of research. I also chose educational sciences because the journal's approach is interdisciplinary and my work can be located there. Nevertheless, I have replaced "lessons" with "education" in the title, as this is less specifically related to teaching.

The use of the word ‘student’ could be sharpened throughout the manuscript; that is, make it clear when you mean the pre-service teachers or student teachers and when you mean pupils. For instance, on line 486, do you mean pre-service teachers who assessed the pupils’ texts (i.e., “named by the students in the student text”). Or in Figure 3, instead of writing about ‘students’ experiences,’ you could write about ‘pre-service teachers’ experiences.’

I have also adapted the terminology here.

pupils' & pre-service teachers

I have also adapted the title of Fig. 3.

In terms of the abstract, I think that the sentence on lines 15–16 breaks the otherwise good flow of the abstract (i.e., “The study sheds light on the importance of diagnostic and feedback skills for (prospective) teachers and the current state of research in this area.”) That is, consider moving the sentence to the beginning or end of the abstract. On lines 16–17, ‘students’ are mentioned twice; therefore, the reader does not necessarily know whether you mean the student teachers or pupils. Thus, consider naming the (university) students as pre-service teachers in that sentence. On line 8, the problems of argumentation are linked both to teachers and students, but for some reason the challenges are only linked to teachers later on line 9.

I have moved the sentence to the beginning.

I also changed the names for pupils and student teachers and mentioned that there are challenges for both groups of actors.

The research questions are proper, and I find it good that, for instance, ‘geography’ and ‘student teachers’ are repeated in all the three questions. However, they are a bit difficult to understand, so consider if there would be a way to make them sharper and more pronounced. In addition, I am not sure if your purpose is to repeat the research questions in exactly the same format later in the manuscript; that is, research question 2, for instance, is in a different format on pages 2 and 15.

Many thanks for the feedback. I have adapted and standardised the research questions.

I have tried to formulate them more clearly:

„1. What experiences and attitudes do pre-service geography teachers have towards diagnosing pupils' argumentative competences in geography education?

2. How do pre-service geography teachers practice diagnosing pupils' argumentative competence based on their argumentative texts?

3. How do pre-service geography teachers give feedback on pupils' argumentative texts?”

In terms of Figure 1, could all four steps be mentioned in the text, as well, as clearly as possible. Especially how ‘feedback’ and ‘support’ link with each other. In addition, in the text, there should be a reference to figures before the figure itself (now figures come first and references follow). For the reader, it would be more logical to read the text first and then look at the figure when indicated by a reference in the text. Edit the issue with all the figures and tables.

Thanks for the hint. I have added a short description. This was not done before, as the article is part of my dissertation and these steps were already covered in detail in the previous article, which I also refer to.

I have also checked again whether it is more logical to read the illustrations or the text first. I would like to stick to the fact that in most places wine introduction for the figure comes before the figure and then the concrete description.

In terms of section 2, ‘language,’ ‘argumentation,’ and ‘structure’ could be elaborated in more detail because you refer to them many times in result section.

I have rewritten the theory section on argumentation skills. In doing so, I have explained the criteria grid, which I refer to throughout the article, in more detail (language, argumentation, structure), removed specific requirements from Germany and added international literature.

In terms of section 3, there are quite many details and information for the German readers; that is, for someone who is not familiar with the German context, the section is quite difficult to read. Therefore, I suggest that you will edit section 3 keeping your international audience on mind. I think that you could represent the studies on a more general level. In addition, consider if section 3 could come before section 2; that is, could section 3 work as section 2 and section 2 as section 3? In addition, could the sentence on lines 415–417 (i.e., “The current state of research shows many different correction habits and evaluation criteria among teachers: there are no general standards for text evaluation and teachers evaluate very differently regarding the criteria and proceed differently”) be moved to the beginning of the section in question? 

I have adapted, summarised and shortened the current state of research.

I have emphasised the international studies and shortened the German studies.

Nevertheless, I have decided to first explain the theoretical foundations in the form of section 2 and then present the current state of research, because the transition to the methodology section is then smoother and the definitional foundations have already been laid.

In terms of section 4, what do you mean by a ‘planned status diagnostic’ (line 459) and a ‘planned-for-interaction assessment’ (line 461)? These should be described in more detail. In addition, you could state that the 8^th grader participated in the study voluntarily and give information about how you selected this particular pupil to participate in your study in the first place (line 450). It would be interesting if you could add the text made by the 8^th grader to your manuscript, or if you could add an example of the assessment done by one of the pre-service teachers. You should also describe in section 4 how you refer to your participants later in the manuscript; for instance, what does ‘SI01212’ mean on line 550.

The pupil wrote the text as part of a homework assignment. It in no way served as a basis for assessment for the teachers. The researchers have no direct reference to the student text and it was sent to us anonymised as a sample text. We chose this text from among several student texts because it was one of the only ones that was comprehensible, readable and of an appropriate length. It shows a balance of strengths and weaknesses of an argumentative text and contains aspects that can be assessed in every area (language, structure, argumentation).

I have refrained from including the student text in the manuscript, as it is not feasible to translate this German text with all its sources of error into English with the same meaning. In addition, the focus should not be on the student text and the specific assessment and correction (this already happens in the previous study), but on the diagnostic steps of the students. Furthermore, it is not about the students' competences and how correctly they have assessed, but only about the design of the process. By including the text, there is a chance that this could be misunderstood by the reader.

Examples of the students' assessments are included in the results section and are therefore not anticipated in the description of the methodology.

I have added the description of the codes in the "sample" section.

In terms of the results, there is an ontological problem there that you are expecting your participants to “evaluate one’s own teaching” (line 870), but how could they execute this or even come up with this thought because they have not been teaching or giving the instruction for the 8^th grader themselves.

Thanks for the comment. I have added this to the section "Methodological criticism and limitations of the study” weitergehend beschrieben und kritisiert.

Comments

Correction

·      Check the tense on line 69 (i.e., could it be “research questions are answered” or “research questions will be answered” instead of “were answered”).

·      In an article, there are no ‘chapters’ but ‘sections’ (e.g., lines 71 and 175).

·      On line 73, a ‘theoretical model’ is mentioned; thus, would it be worthy to refer to the researcher whose model it is.

·      Is it necessary to state “and many more” in a reference on lines 115–116? Check the same issue on line 918; that is, delete “according to” from the reference.

·      The quotation on lines 171–173 is difficult one to understand. Could this be written with your own words instead of using a quotation (especially because the original research is in German and not in English)? I would suggest not to use quotations in cases in which the original research is in German throughout the manuscript.

·      I am not sure what do you mean by the model serving “as an orientation for geography teachers” (lines 182–183). As an orientation for what (cf., line 197)?

·   

   What does the star symbol (*) stand for on lines 311 and 584?

·      Period at the end of a sentence is missing on line 317. Period is also missing in Table 2 in the description of ‘advising.’

·      Is it necessary to state the effect size on line 346?

·      What does the backslash mean in ‘Parr/Timperley’ (line 347) and in ‘Cho/MacArthur’ (line 367)? Do you mean ‘Parr and Timperley?’

·      I would suggest to better link the ’local’ and ‘global’ aspects (line 363) with the ‘structural’ and ‘content-related’ aspects (line 362) mentioned in the previous sentence. You continue to describe the local and global texts on a footnote on page 11, but instead of describing these concepts in more detail in a footnote, you could do it on line 363.

·      Is ‘grading are largely based on written grading’ (line 393) correct (i.e., grading is based on grading)?

·      I do not know what requirement areas 1, 2, and 3 are (lines 402–404). This is quite specific information for the German readers; however, your audience is international.

·      I do not know what ‘Hamburger Schreibprobe’ is (lines 423–424). This is quite specific information for the German readers; however, your audience is international.

·      Is ‘argumentative pupil texts’ grammatically correct on lines 425, 433, 598, and 845 (cf., main title)?

·      The word ‘evaluation’ is repeated three times in one sentence on lines 424–425; that is, consider revision. Likewise, the word ‘feedback’ is repeated three times in one sentence on lines 827–828; that is, consider revision.

·      Who do you mean by ‘geography students’ on line 427; that is, pre-service teachers or school pupils?

·      Instead of using abbreviations, ‘Fig. 2’ could be written ‘Figure 2’ (page 10 and throughout the manuscript).

·      On line 468, you refer to ‘Table 2,’ but it should be ‘Table 1.’

·      Is ‘high school’ correct in the footnote on page 10?

·      In Table 1, you use the word ‘student,’ but you could clarify this using either ‘student teacher’ or ‘pre-service teacher’ instead. In addition, ‘he/she’ structure is a little bit clumsy, which could be avoided by simply using ‘they’ (e.g., “which shows that they have reviewed the text product…”).

·      Make sure you name the school levels correctly on line 512; that is, how do lower secondary, intermediate secondary, and secondary schools differ from comprehensive school?

·      On line 523, instead of writing ‘the following figure provides,’ you could name the figure accurately; that is, ‘Figure 3 provides.’ Check the same issue on line 779.

·      References on line 551 should be marked ‘[31-33, 35, 39].’ Check the same issue on lines 929 and 930.

·      Instead of referring to the criteria grid by Steingrübl and Budke on line 627, this could already be done in the theoretical background on line 233.

·      In Figure 5, ‘criteria for positive aspects’ is repeated twice.

·      Why is Figure 8 named as Figures 8a and 8b and not as Figures 8 and 9? In addition, these figures are too difficult to read, one reason being the color choices. In Figure 8a (that should be named Figure 8), for instance, the order of the colors in the legend does not follow that of the diagram itself. In terms of Figure 8b (that should be named Figure 9), the use of the diagram in representing the results is not convincing—especially because you state that “there are no clear characteristics” on line 775. Consider what it is that you want to elaborate with Figure 8b and try a more suitable manner.

·      The 22 criteria deriving from the literature (line 917) should be described in theoretical background.

·      It is confusing for the reader to refer to a ‘three-step diagnostic cycle’ (e.g., line 956) when you have five steps in your diagnostic cycle (Figure 2).

Changed into „will be answered“

“Chapter” was changed into “section”

I created the model based on the model by Aufschnaiter, Münster & Beretz (2018:384). I have therefore decided to mention it in the text as follows "theoretical model (based on Aufschnaiter, Münster & Beretz (2018:384)) "

changed

I changed the quotation into:„In addition to interdisciplinary principles, diagnostic procedures in geography education also include didactic considerations: These provide criteria and guiding questions for diagnosis, which can already refer to aspects relevant for support and allow a categorisation of interpretations regarding the causes of the pupils' competence level and can be used to derive consequences in the form of support measures (Beretz, 2020:27).”

Changed into:
“to guide geography teachers in diagnosing pupils' argumentative writing skills in geography lessons.”
“in terms of existing pupil competencies”

Deleted

Added

Yes, changed

Yes, changed

I would prefer to keep the facts separate at this point: One section is about the current state of research and the survey grid should at best also be able to support teachers or researchers, so I would like to explain this in more detail in the footnote. Nevertheless, I have followed your advice and linked the content, linguistic and structural aspects better with the local and global aspects of the current state of research.

Deleted

Deleted

Deleted

Yes

Changed

Changed to pre-service teachers

I changed everything to “Figure”

Thanks, were changed.

Yes, high school is right.

Thanks for the tip. I have adjusted the table

I have changed this to 'lower secondary teaching'.

Changed

Changed

Changed, added to section 2

Changed

I have changed the names of the graphs to Figure 8 and Figure 9. Previously they were called 8a and 8b because they belong together and show the same criteria: One shows the level and the other the order in which the criteria were applied in the feedback text. The order of the legend: This is read from left to right, so it is the same order as in the graph. I have adapted the choice of colours. I have also adapted Diagram 9 to make it easier to understand.

Added in section 2

Changed into “5 steps”

Comments

Corrections

Use keywords not present in the Title.

Thanks!

I changed the keywords into:

diagnosis in geography education, pre-service teachers behavior, text assessment, written feedback, argumentation evaluation  

Please provide a clear rationale for the study. In your Introduction (before presenting your research questions), discuss the need to explore the diagnostic and feedback behavior of university students regarding argumentative pupils' texts in geography lessons.

Thanks for the hint. I thought that the introduction to the topic, starting with the relevance of the argumentation for geography lessons and the lack of competence of students and teachers, as well as the resulting goal of individual support, was sufficient to justify the research. However, I have added another sentence: „In order to be able to start concrete implementation of support in university didactics, it is necessary to research the current status of pre-service teachers and their current practices, as well as their attitudes towards diagnosis and feedback on argumentative student texts.”

Under the Methods section, provide a separate discussion for the Data Gathering Procedures, Research Instruments, and Data Analysis. The discussion for the Research Design must focus on what approach grounded your study, not the procedures. Some discussion under your "2.0" is more appropriate for the Methods section.

I have added a sentence about the research design and included it as a separate chapter. However, I would still like to present the research questions with the survey and evaluation methods as before, as otherwise the flow of reading would be disrupted and the article would become even longer due to the reference to a new separate chapter on evaluation methods.

In discussing the study samples, kindly expound the inclusion (and exclusion) criteria. Why were they chosen, and how did you choose them?

The sample was not purposively selected, but arose from the opportunity to conduct the study in a bachelor's seminar. The students were not specifically trained in diagnosis in advance.

I changed this as follows „This is a convenience sample, which means that the sample was selected neither purposively nor randomly, but solely on the basis of accessibility, practicality and voluntariness. Therefore, there were no specific exclusion criteria for selection.

The test participants were 20 pre-service geography teachers with an average age of 23.5 years. Each participant studied geography at lower-secondary level and had already studied the subject for an average of 4.5 semesters. The study took place in a bachelor's programme at University of Cologne (Germany) on planning geography lessons. Eight pre-service teachers are studying economics and politics as a second subject, five are studying German, five are studying biology and two are studying history.

The study was anonymous, voluntary and non-evaluative. No content relevant to the study was taught beforehand in the seminar, and the pre-test (see next section) was used to assess the participants' previous knowledge and experience.”

Do NOT add citations in the Conclusion section. This section should highlight new meanings that can be derived from your results, which MUST answer your research questions.

I rewrote this part and answered the research questions more precisely and focussed on my study.

Add a paragraph recommendation based on your conclusion. Indicate the limitations of the current work with your suggestions for future directions.

I rewrote this part and answered the research questions more precisely and focussed on my study

Strictly follow the citation and referencing style set by the journal. 

Thanks, I have revised this

The research design worries me.

Firstly, it is stated in the abstract that the study uses “a qualitative research design.” However, the results are about percentages of how many of the twenty pre-service teachers executed a certain performance in the study. I personally am not convinced that this is the best possible methodology to analyze the dataset when the number of participants is small. 

Thank you again for your quick feedback. I am very grateful that you are so interested in my research and take the time to ensure that the article and its contents are as comprehensible as possible for readers. I will now try to incorporate the comments so that the article will be accepted in its new form.

I have decided to present the results as a percentage, as it makes sense to me to present relative frequencies in a qualitative context. This makes it easier for the reader to recognise relationships.

Nevertheless, I have changed the text in some points and used absolute frequencies when it comes to the number of participants.

I actually only found the percentages relating to the teachers in Figure 5 and removed the percentile ranks there. Otherwise, I always write "4 out of 20 PS teachers" in the body text and therefore no percentages.

When it comes to the number of criteria used, for example, there is a higher number, which is why I have kept the presentation of the criteria with relative frequencies here.  

Secondly, it is stated in the manuscript that teachers should use criteria grids in assessment. However, the authors did not guide the participants to use any sort of criteria grid in the study. I find this to give a rather contradictory example for the readers. In addition, the pre-service teachers were not familiar with the diagnostic cycle either, which however has an important role in the manuscript.

There is a misunderstanding in this context. The aim is to collect results that show which criteria are not yet recognised and where there is still a need for action to raise awareness. The fact that the students should use criteria grids is ultimately only the result, a suggestion, so to speak, as to how they can strengthen their diagnosis. The same applies to the use of the diagnostic circle. I designed the study as an experiment so that I could gain insights into the current situation. In further research, I would then conduct quantitative research where I hypothesised that using the criteria grid and the diagnostic circle could lead to a more complete diagnosis. Nevertheless, I felt it was initially relevant to research the current status, as it cannot be assumed that these tools are used directly in everyday university or school life.

What I did not describe in the article is that the students were given a final task after the examination in which they received the criteria grid and were able to revise their diagnosis again. This was only dealt with briefly, but the initial results show that they naturally saw more criteria on the basis of this, particularly in terms of content. This would then be the outline for the next article. This article is already very long and that's why I wanted to put these results, which I thought were too unscientific, in the next article, where it would also fit better thematically when it comes to evaluating the support formats.

I can understand your point of view, which is why I have explained this even more clearly in the methodological critique. I hope you like it better now.

„The theoretical models, such as the criteria grid and the diagnostic cycle, were not shown to the participants beforehand or given to them to help them complete the tasks. This could have altered the results. In this research project, the models only served as part of the evaluation methodology and as a preview of possible support measures for prospective geography teachers. The research design can be criticized in that there is no comparison between a group of subjects who are diagnosed with knowledge of the criteria grids and the diagnostic cycle and a group who are not. However, this comparison would have addressed a further research question that would have evaluated the models. The purpose of this basic experimental research was to present the current state of diagnosis and feedback design, and to provide a starting point for quantitative, hypothesis-driven research into the use or non-use of the models in relation to behavioral practice.”

“ As this was an experimental, unnatural setting with an unknown pupil, it would be interesting for future research to see how the pre-service teachers would behave in a natural setting and with support and theoretical instruction on the diagnostic cycle and assessment criteria.“

Thirdly, in conclusions it is emphasized that the study in hand is “an initial exploration” and “an experimental, unnatural setting.” This could already be stated in the very beginning of the manuscript so that the reader would be aware of it. 

Instead, the study design could be, for instance, about a comparison in terms of how the pre-service teachers assessed the text with and without the help of the five-step diagnostic cycle. At such, the current research design and the conclusions drawn from the dataset might be too ambitious. 

See above.

I also mentioned directly in the introduction that this is an experimental design, which is intended to show initial findings and prepare for a quantitative comparative study.

I have added the following sentence to the introduction „ The research design is a qualitative experimental setting, the initial aim of which is to gather first impressions on the previously unexplored topic of diagnosis and feedback on argumentative texts in the context of geography education.“

In addition, I am still not convinced that pupils aged 13–14 years are in high school (page 11). And it sounds strange that the respondents created the pseudonyms themselves (page 14). What sort of instruction were they given in order to do this? Did they have to use two letters and four numbers in their pseudonyms? What I mean is that it would be simpler for the reader if you used pseudonyms such as “respondent 1,” “respondent 2,” etc. That is, the use of “SA,” “MI,” “EL” etc. raises more questions than answers from the perspective of the reader. In addition, consider is it necessary to mention geography education in the main title

Yes, I have researched this several times and now, so that it is completely unambiguous, I have rephrased it as:

„The pupil is a 14 years old and in the 8th grade of a secondary school equivalent to a high school or grammar school.“

Thank you for pointing this out, I agree and have adjusted the abbreviations accordingly.

It is essential to use the geography reference in the main title, as I am concerned with raising awareness in geography didactic research (and of course the geographical audience) that these actions must also take place there and that specific specialist skills are necessary for this and that this is not only relevant in the language subjects.