Innovation Off the Bat: Bridging the ChatGPT Gap in Digital Competence among English as a Foreign Language Teachers

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

A very interesting and promising area of research, addressing the key issues.

A very-well organized and well-written piece of paper which can contribute to the use of AI tools and ChatGPT in the EFL context.

Author Response

Dear Reviewer,
Thank you so much for this warm review. We appreciate it.

Reviewer 2 Report

Comments and Suggestions for Authors

The small and geographically limited sample size raises substantial concerns regarding the generalizability of the findings. With a restricted and possibly non-representative sample, it is difficult to draw conclusions that can be applied to broader populations. This limitation severely undermines the external validity of the study.

The short duration of the intervention may not adequately capture the long-term effects or sustainability of the observed improvements. Without evidence of lasting impact, the practical significance and real-world applicability of the findings are questionable.

The lack of a control group is a critical methodological flaw. Without a control group, it is impossible to definitively attribute the observed changes to the intervention. This omission compromises the internal validity of the study and raises doubts about the causal relationship between the intervention and the outcomes.

The study acknowledges that pre-service teachers were hesitant to incorporate ChatGPT into Science education due to the model's training data cutoff in September 2021, which could result in outdated or erroneous information. Although this limitation might be less critical in the context of English language teaching-learning, it nonetheless highlights potential biases and inaccuracies inherent in GenAI tools. This concern further complicates the study's reliability and raises questions about the broader applicability of the findings.

Given these substantial limitations, the study's conclusions are likely to be flawed or overstated. Publishing the article in its current form could mislead readers regarding the effectiveness and applicability of the intervention. Therefore, it is recommended that the article not be published until these limitations are addressed and the study design is improved to provide more robust and generalizable findings.

 

Comments on the Quality of English Language

The small and geographically limited sample size raises substantial concerns regarding the generalizability of the findings. With a restricted and possibly non-representative sample, it is difficult to draw conclusions that can be applied to broader populations. This limitation severely undermines the external validity of the study.

The short duration of the intervention may not adequately capture the long-term effects or sustainability of the observed improvements. Without evidence of lasting impact, the practical significance and real-world applicability of the findings are questionable.

The lack of a control group is a critical methodological flaw. Without a control group, it is impossible to definitively attribute the observed changes to the intervention. This omission compromises the internal validity of the study and raises doubts about the causal relationship between the intervention and the outcomes.

The study acknowledges that pre-service teachers were hesitant to incorporate ChatGPT into Science education due to the model's training data cutoff in September 2021, which could result in outdated or erroneous information. Although this limitation might be less critical in the context of English language teaching-learning, it nonetheless highlights potential biases and inaccuracies inherent in GenAI tools. This concern further complicates the study's reliability and raises questions about the broader applicability of the findings.

Given these substantial limitations, the study's conclusions are likely to be flawed or overstated. Publishing the article in its current form could mislead readers regarding the effectiveness and applicability of the intervention. Therefore, it is recommended that the article not be published until these limitations are addressed and the study design is improved to provide more robust and generalizable findings.

 

 

Author Response

Comment 1: The small and geographically limited sample size raises substantial concerns regarding the generalizability of the findings. With a restricted and possibly non-representative sample, it is difficult to draw conclusions that can be applied to broader populations. This limitation severely undermines the external validity of the study.

Response 1: Thank you for the comment. However, we distinctively acknowledged this limitation in the Limitations section (page 12, lines 458 and 459). Hereinafter, we refer to the revised version of the manuscript.

Comment 2: The short duration of the intervention may not adequately capture the long-term effects or sustainability of the observed improvements. Without evidence of lasting impact, the practical significance and real-world applicability of the findings are questionable.

Response 2: We distinctively acknowledged this limitation in the Limitations section (page 12, lines 459 and 460).

Comment 3: The lack of a control group is a critical methodological flaw. Without a control group, it is impossible to definitively attribute the observed changes to the intervention. This omission compromises the internal validity of the study and raises doubts about the causal relationship between the intervention and the outcomes.

Response 3: We distinctively acknowledged this limitation in the Limitations section (page 12, lines 460 - 462).

Comment 4: The study acknowledges that pre-service teachers were hesitant to incorporate ChatGPT into Science education due to the model's training data cutoff in September 2021, which could result in outdated or erroneous information. Although this limitation might be less critical in the context of English language teaching-learning, it nonetheless highlights potential biases and inaccuracies inherent in GenAI tools. This concern further complicates the study's reliability and raises questions about the broader applicability of the findings.

Response 4: We mentioned the concerns in the Limitations section (page 12, lines 465 - 468).

Comment 5: Given these substantial limitations, the study's conclusions are likely to be flawed or overstated. Publishing the article in its current form could mislead readers regarding the effectiveness and applicability of the intervention. Therefore, it is recommended that the article not be published until these limitations are addressed and the study design is improved to provide more robust and generalizable findings.

Response 5: Dear Reviewer, this study represents one of the first academic efforts to obtain quantitative evidence on teachers' literacy in employing GenAI, as well as qualitative insights into their perceptions of this technology. Given the pioneering nature of this research, the use of a small sample is justifiable, especially when (a) the participants are educators, who tend to have busier schedules than students, and (b) the study involves an intervention rather than a survey.
For reference, please see this study conducted on 12 subjects, published in Education Sciences two weeks ago:
https://www.mdpi.com/2227-7102/14/8/805
While it would have been easier to follow the common approach of collecting opinions on the topic using a Likert scale, or conducting a case study (such as this one: https://www.castledown.com/journals/tltl/article/view/1142), we chose to go beyond that. We made an extra effort, and we kindly ask that you acknowledge and respect this.
Thank you for your comments.

Reviewer 3 Report

Comments and Suggestions for Authors

 

Dear Author(s), I have read your manuscript titled "Innovation Off the Bat: Bridging the ChatGPT Gap in Digital Competence among EFL Teachers" and it presents a timely and relevant exploration of integrating ChatGPT into EFL teaching. While the study offers valuable insights, major revisions are necessary to enhance the clarity, detail, and rigor of your solid research.

 

 

Abstract

The abstract should explain the nature of the ‘guided experimental implementation’.  It might also be useful to explain ‘pre-engineered prompts’

 

Introduction

It is strongly suggested the authors divide it into a separate introduction and a separate literature review.  While the first couple of paragraphs effectively identify the gap, the lack of a separate literature review makes it ineffective. By separating the two, the authors will be able to better position the current study within the existing body of research

 

The introduction briefly mentions the technological pedagogical content knowledge (TPACK) framework. Expanding on this framework and how it guides the study would provide a clearer theoretical foundation.

 

 

Research Questions

‘the chatbot’ – the authors need to be specific

 

Participants and Data Collection

Include the ethics number

 

Participant selection

Provide more detail on how participants were selected beyond availability and willingness. What specific criteria were used, and how do these criteria ensure a representative sample? Explain why the gender distribution is heavily skewed (22 females and two males) and discuss any potential implications of this imbalance.

 

Intervention details

Is it possible to include a flowchart to illustrate the intervention?

 Include more detail on the online introductory workshop. What specific topics were covered, and how were participants prepared for the intervention?

 

Data collection

\Explain the custom-designed assessment tool used to measure ChatGPT integration proficiency. What specific metrics or criteria were assessed? The qualitative data collection through a self-reflection tool should be described in more detail. How were responses analyzed, and what themes emerged?

 

Limitations

This should more explicitly acknowledge the study's limitations, such as the small sample size and potential biases in participant selection. Discuss how these limitations might affect the generalizability of the findings and what steps could be taken to address them in future research.

 

Conclusion

Add implications to this section: Conclusion and Implications

 

Discuss the modest, practical significance in more detail. Why might the improvements be modest, and what factors could influence this?

 

Suggest areas for future research, particularly studies that could build on the findings of this research and address its limitations. Use a separate sub-heading - Future Research

 

Author Response

Comment 1: Abstract
The abstract should explain the nature of the ‘guided experimental implementation’.  It might also be useful to explain ‘pre-engineered prompts’

Response 1: Dear Reviewer, we believe that the abstract already outlines the nature of the intervention. However, we replaced 'pre-engineered' with 'researcher-designed' for clarity.

Comment 2: Introduction
It is strongly suggested the authors divide it into a separate introduction and a separate literature review. While the first couple of paragraphs effectively identify the gap, the lack of a separate literature review makes it ineffective. By separating the two, the authors will be able to better position the current study within the existing body of research.

Response 2: Done.

Comment 3: The introduction briefly mentions the technological pedagogical content knowledge (TPACK) framework. Expanding on this framework and how it guides the study would provide a clearer theoretical foundation.

Response 3: Dear Reviewer, thank you for pointing this out. The manuscript has been revised to address this topic (pages 2 and 3, lines 80 - 98).

Comment 4: Research Questions
‘the chatbot’ – the authors need to be specific

Response 4: Fixed (page 3, line 126).

Comment 5: Participants and Data Collection
Include the ethics number

Response 5: Done (page 3, line 148).

Comment 6: Participant selection
Provide more detail on how participants were selected beyond availability and willingness. What specific criteria were used, and how do these criteria ensure a representative sample? Explain why the gender distribution is heavily skewed (22 females and two males) and discuss any potential implications of this imbalance.

Response 6: The revised manuscript describes two inclusion criteria (page 3, line 143 - 145). The gender skew is due to the global predominance of females in the Humanities disciplines, including foreign languages Anyway, our research needs to be reiterated by further researchers across multiple disciplines, and we believe there will be gender balanced samples.

Comment 7: Intervention details
Is it possible to include a flowchart to illustrate the intervention?

Response 7: Provided now (page 4, between lines 171 and 172).

Comment 8: Include more detail on the online introductory workshop. What specific topics were covered, and how were participants prepared for the intervention?

Response 8: Detailed now (page 4, lines 154 - 164).

Comment 9: Data collection
Explain the custom-designed assessment tool used to measure ChatGPT integration proficiency. What specific metrics or criteria were assessed? The qualitative data collection through a self-reflection tool should be described in more detail. How were responses analyzed, and what themes emerged?

Response 9: Dear Reviewer, we believe the details are provided in the manuscript (pages 6 and 7, 232 - 280).

Comment 10: Limitations
This should more explicitly acknowledge the study's limitations, such as the small sample size and potential biases in participant selection. Discuss how these limitations might affect the generalizability of the findings and what steps could be taken to address them in future research.

Response 10: The Limitations section clearly acknowledges the small sample size (page 12, lines 458 and 459). We also suggest that future research should use larger samples (page 12, lines 471 - 473).

Comment 11: Conclusion
Add implications to this section: Conclusion and Implications

Response 11: Done (page 12, 477).

Comment 12: Discuss the modest, practical significance in more detail. Why might the improvements be modest, and what factors could influence this?

Response 12: Added (page 11, lines 421 - 430).

Comment 13: Suggest areas for future research, particularly studies that could build on the findings of this research and address its limitations. Use a separate sub-heading - Future Research

Response 13: Done (page 12, lines 468 - 476). Thank you for your comments.

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

Thank you for informing me about the revisions. I have reviewed the revised manuscript, and I am pleased to confirm that the changes meet the required standards. The paper is ready for publication.

Comments on the Quality of English Language

Thank you for informing me about the revisions. I have reviewed the revised manuscript, and I am pleased to confirm that the changes meet the required standards. The paper is ready for publication.

Author Response

Comment 1: "Thank you for informing me about the revisions. I have reviewed the revised manuscript, and I am pleased to confirm that the changes meet the required standards. The paper is ready for publication."

Response 1: Dear Reviewer, thank you so much!

Reviewer 3 Report

Comments and Suggestions for Authors

Thank you for submitting your revised manuscript. I appreciate your effort in revising your manuscript based on previous feedback; it is now more robust. However, I have identified a few more areas that should be further revised to clarify and develop ideas.

 

Theoretical Framework

The authors have situated their study within the TPACK framework; however, the connections between the framework, RQs, and design should be made more explicit. Elaborate on how the TPACK dimensions align with your ‘Operationalisation’ of AI literacy. The literature review should also be more critical – the authors need to engage with the literature more critically. There should be a more explicit discussion of contradictory findings or gaps in the existing literature.

 

 

Methodology

Provide more details on the purposive sampling strategy. What criteria were used to determine participants’ availability and willingness? The authors should also discuss any potential bias in the sample selection.

The validity and reliability of the CIPA are central to your findings. Provide more information on the pilot testing process and how the feedback was incorporated. Report any inter-rater reliability measures for scoring the open-ended responses.

Justify the use of matched pairs analysis. Given the small sample size, discuss the appropriateness and limitations of the statistical tests employed.

 

Discussion

 

How do your results compare to prior studies on AI in language education? What are the unique contributions and limitations of your study?

 

 

Author Response

Comment 1: "Theoretical Framework. The authors have situated their study within the TPACK framework; however, the connections between the framework, RQs, and design should be made more explicit. Elaborate on how the TPACK dimensions align with your ‘Operationalisation’ of AI literacy. The literature review should also be more critical – the authors need to engage with the literature more critically. There should be a more explicit discussion of contradictory findings or gaps in the existing literature."

Response 1: Dear Reviewer, the section was amended (page 2, lines 51 - 56; page 3, lines 103 - 105). Three additional papers were cited.

Comment 2: "Provide more details on the purposive sampling strategy. What criteria were used to determine participants’ availability and willingness? The authors should also discuss any potential bias in the sample selection."

Response 2: The details are now provided on pages 3 and 4, lines 145 - 154.

Comment 3: "The validity and reliability of the CIPA are central to your findings. Provide more information on the pilot testing process and how the feedback was incorporated. Report any inter-rater reliability measures for scoring the open-ended responses."

Response 3: Done (page 7, lines 261 - 265; page 8, lines 277 - 280).

Comment 4: "Justify the use of matched pairs analysis. Given the small sample size, discuss the appropriateness and limitations of the statistical tests employed."

Response 4: Done (page 8, lines 300 - 305). One additional paper was cited.
 
Comment 5: "How do your results compare to prior studies on AI in language education?"

Response 5: As highlighted in the Introduction of our study, the body of research on the integration of GenAI in education, particularly in the context of language education, is still in its infancy, with a significant lack of objective experimental evidence. Thus, our findings may not directly compare with prior studies due to the difference in research design and objectives.

Comment 6 : "What are the unique contributions and limitations of your study?"

Response 6: Added (page 13, lines 488 - 493).
Thank you!