Assessing Teachers’ Capabilities to Work with Models and Evaluate Results in the Context of a Complex and Authentic STEM Problem
Round 1
Reviewer 1 Report
Comments and Suggestions for AuthorsThe abstract could be more cut down as it reads more like an introduction and context right now. For example, starting at Line 14, you could omit the prompt and the specific context of the experiment. Also, don’t need to cite the qualitative method in the abstract. Mainly, it could be shortened.
Both in the abstract and the introduction, “practice turn” is label with “so-called” and not really defined in terms of the policy context. I would recommend omitting “so-called” and being more general that “modern educational paradigms have transitioned from…”
Also, it is not clear from the quote what the authors mean about “simply teaching science ideas.” This could be explained more clearly what the authors intended.
Starting at Line 56, I am not sure why the paper discusses the lack of transferability of complex and authentic problems. Seems out of place and devalues the purpose of your study. This relates to the literature review on transferability in Section 2.1.2. It is not clear why the author(s) are talking about it in terms of student learning rather than for the purposes of teaching, which is a different type of learning for problem solving more related to pedagogical content knowledge. The section does not seem necessary or relevant to the study.
At Line 65, no need to discuss iterative process of Kuckartz until methods. It needs to be explained and you need to give a rationale as to why that approach, which is better suited in methods section. Relatedly, there is no literature cited or explained as to why that approach is the best to address this study.
There is a literature review for both complex and authentic problems. Although the authors describe the problem, it is not explicitly connected to the literature on how the chosen problem is complex and authentic according to the definitions, particularly Exercise 1, which is typical of a word problem and does not seem to be authentic in the sense argued by the authors. It is not clear why these problems can be used to evaluate whether teachers can solve complex and authentic problems. You are basing your results on one type of mathematical problem. This relates later to the results where the findings seem overstated for the context, participant demographic, and choice of problem. Really it seems that you found that teachers could not solve this TYPE of problem. And, again, why is this important to know and address? Also, why say you are studying STEM, when you know you are not? It is more about mathematics and computer science, which is actually more about applied mathematics than pure mathematics.
Section 4.1 is unclear as to the methods of this study. It is mentioned that students had issues solving the problems, and that was used to argue the selection of mathematically strong teachers. However, that does not seem to have any methodological connection for that rationale. Teaching teachers how to teach problem solving is very different than assessing student understanding of a problem. I suggest omitting Section 4.1 and just stating that you chose well educated secondary school teachers to avoid having to teach the content and focus more on the process of teaching problem solving.
In the selection of the participants, there are several teachers that did not meet the criteria. So, why have any criteria for selection if you are not going to adhere to it? More than half of the sample did not meet the criteria. Or at least give a different rationale for selection.
Research questions should be moved closer to the introduction so that we know what you will study and have a clearer idea of the theoretical perspectives chosen. The current research question focuses on the educators’ abilities and evaluation of the model, which is focused more on teacher learning than what I thought the study was going to focus on, which I thought would be teacher learning about problem solving so they can teach it. Also, there are way too many questions for one study. Some of the questions are not research questions but more methods. For example, “How many teachers created a satisfactory evacuation estimates as solutions?” does not require investigation or analysis beyond assessing their mathematical solutions. Also, it is a closed question. It would help to go back through the first 9 pages and focus more on teacher learning and less about student learning for this reason. Mainly, why should we care that a teacher is able to solve this problem? How is that going to improve instruction? What do you need to investigate to determine your answers to those types of questions? In addition, Section 5.2 is discussing students. Was this supposed to be about the teachers? In this section, it seems that the research question could really be something like, “What skills did participants use to solve a complex an authentic problem?” But again, why do we care about that?
The authors claim they used a mixed methods approach, but they did not because the methods answered different questions and not overall question that could be answered using both a quantitative and qualitative perspective. It seems a misapplication of what a mixed methods study is supposed to be. Even the “quality of solutions” was evaluated quantitatively and against a null hypothesis.
Overall, there is a lot of information, but I do not see a coherence between the literature of STEM teacher education, research questions, methodology, analysis, and claims. It seems that there is an abundance of pieces, but not a cohesive narrative that goes deeply and contributes to the needs of STEM teacher education. This can be addressed by really narrowing the focus and determining which data are needed to answer a question about teacher learning and why that is necessary to address for STEM students’ learning needs and opportunities.
Comments on the Quality of English LanguageThere are minor errors here and there. These are some examples:
Omit term “so-called” from manuscript.
Do not capitalize improper nouns and terms (e.g., working with models, complex and authentic problems)
Line 61: missing a period after “…on complex STEM problems.”
Line 87: consider rephrasing “students as problem solvers” to “positioning students as problem solvers” or something similar.
Line 355, word should be “incentivize.”
Line 357, should be “a P”
Author Response
Dear Reviewers,
thank you for your constructive and comprehensive feedback! We were able to put quite a lot of work into the article and are confident that it has improved quite a lot!
The main thing we changed was the scope and focus of the article. The original draft tried to do too many things and went in too many directions. We addressed this by focusing clearly on the assessment of capabilities promised in the title. As such, the structure of the paper was greatly simplified, the length was reduced, readability improved, and the goals are described more precisely.
The evaluation of the argumentation quality now has a solid theoretical basis: The SOLO-taxonomy was used to assess the quality (in form of the structural complexity) of the evaluation exercise. The taxonomy itself is outlined in Section 2.2.5.
We also tried to make it clearer that the tasks indeed rely on activities that educators are currently expected to have. Sections 3.4 has been extended with quotes from curricula. Additionally, we added more related work about the assessment of teachers’ problem-solving capabilities, especially in Section 2.1.1.
Responses to your individual review are attached in the word field.
Best regards,
the Authors
Author Response File: 
Author Response.pdf
Reviewer 2 Report
Comments and Suggestions for AuthorsYou can find my suggestions in the file attached
Comments for author File: Comments.pdf
Author Response
Dear Reviewers,
thank you for your constructive and comprehensive feedback! We were able to put quite a lot of work into the article and are confident that it has improved quite a lot!
The main thing we changed was the scope and focus of the article. The original draft tried to do too many things and went in too many directions. We addressed this by focusing clearly on the assessment of capabilities promised in the title. As such, the structure of the paper was greatly simplified, the length was reduced, readability improved, and the goals are described more precisely.
The evaluation of the argumentation quality now has a solid theoretical basis: The SOLO-taxonomy was used to assess the quality (in form of the structural complexity) of the evaluation exercise. The taxonomy itself is outlined in Section 2.2.5.
We also tried to make it clearer that the tasks indeed rely on activities that educators are currently expected to have. Sections 3.4 has been extended with quotes from curricula. Additionally, we added more related work about the assessment of teachers’ problem-solving capabilities, especially in Section 2.1.1.
Responses to your individual review are attached in the word file.
Best regards,
the Authors
Author Response File: Author Response.pdf
Reviewer 3 Report
Comments and Suggestions for Authors
Dear author,
I hope this review finds you well. I recently had the opportunity to review your manuscript, and overall, I commend you on the well-presented work focused on a topic of significance to the target audience of this journal. Your exploration of teachers' capabilities in constructing models was particularly engaging.
The theoretical section of your paper is broad and captivates the reader's interest. Your introduction seamlessly leads into the research field and articulates the research questions in a compelling and easy-to-follow manner.
The clarity of your research questions, derived from the current state of research, is commendable. Nonetheless, I observed that the sample, encompassing both students and teachers, is well-described but surprisingly doesn't play a significant role in the presentation of results.
Considering the complexity of the manuscript, it might be worthwhile to reconsider the inclusion of research question 11. It might better be deleted without affecting the substance of the manuscript.
To further enhance your manuscript, I offer a list of minor remarks for your consideration:
- The emphasis on research question 5.1-1 appears disproportionate in an explorative study with a limited participant pool. Avoid relying too heavily on numerical data when dealing with a smaller sample size (How many teachers ….?).
- Research question 5.3-9 seems perplexing, considering the limitations of your research design. It appears challenging to explore dependencies between variables based on your methodology.
- On page 11, you introduce two separate scores, namely the point score and the result score. It is unclear how these scores represent distinct information. Clarifying this separation would improve the reader's understanding.
- I could not find table 6 mentioned on p. 12. Ensure that the relevant table is appropriately referred to and located in the manuscript.
- Section 7.6, including the summary, requires improvement. The term "relative amount" in the summary needs clarification for a more coherent presentation.
- The result summary in section 7.7 mentions a lack of reasoning processes. Please verify if your data support this conclusion and provide additional clarity.
- Figure A1 is challenging to follow. Consider revising or providing additional explanations to enhance its clarity.
I believe addressing these suggestions will contribute to the overall refinement of your manuscript.
Best regards!
Author Response
Dear Reviewers,
thank you for your constructive and comprehensive feedback! We were able to put quite a lot of work into the article and are confident that it has improved quite a lot!
The main thing we changed was the scope and focus of the article. The original draft tried to do too many things and went in too many directions. We addressed this by focusing clearly on the assessment of capabilities promised in the title. As such, the structure of the paper was greatly simplified, the length was reduced, readability improved, and the goals are described more precisely.
The evaluation of the argumentation quality now has a solid theoretical basis: The SOLO-taxonomy was used to assess the quality (in form of the structural complexity) of the evaluation exercise. The taxonomy itself is outlined in Section 2.2.5.
We also tried to make it clearer that the tasks indeed rely on activities that educators are currently expected to have. Sections 3.4 has been extended with quotes from curricula. Additionally, we added more related work about the assessment of teachers’ problem-solving capabilities, especially in Section 2.1.1.
Responses to your individual review are attached in the word file.
Best regards,
the Authors
Author Response File: Author Response.pdf
Round 2
Reviewer 1 Report
Comments and Suggestions for AuthorsThis was a huge improvement from the original draft. The organization made it so much easier to read and follow the arguments. I appreciate how mindfully you attended to all the feedback.
Comments on the Quality of English LanguageThere were only small grammatical/spelling errors that I detected. Before publishing, it would help to do one more close read. Otherwise, the authors attended to all the feedback well, making it a much stronger manuscript.
