Harnessing Generative Artificial Intelligence for Digital Literacy Innovation: A Comparative Study between Early Childhood Education and Computer Science Undergraduates

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This study explores how undergraduate students engage with generative AI platforms to generate instructional content, with a pre-post design between two groups (albeit no control groups and only on small samples). It is well-motivated, given the increasing usage of generative AI in education with relatively little pre-post test research published so far. The paper is well-written and it is straightforward to follow the methodology and activity description. Although findings from the comparative approach were limited, both groups appear to have performed well academically and felt fairly comfortable (on average) using the AI tools. This speaks positively to the value of including these tools as part of a rich, broad, and balanced pedagogy and the authors draw out this conclusion nicely. 

 

In general, some of the minor differences between the groups appear to be over-claimed in section 5: "there were notable differences in their experiences and satisfaction with AI tools". However, table 4 has no statistically significant differences in satisfaction for the groups and only 1/3 constructs for user experience are different in table 3. (There are differences in prior experience, but that does not seem to be the phrasing here given the collocation with satisfaction referring to the instructional period). It is possible that the small sample size is limiting the ability to identify meaningful differences between the groups, but if the authors wish to draw such conclusions, they would be best advised to repeat the study with a larger sample. As a result, such claims like the following in the conclusion seem too strong: "By comparing these groups, this research identified factors as educational background and prior technological background that influence user experience, satisfaction, and learning outcomes with AI tools." This may well be the case - indeed it likely is the case to some extent - but the battery of non-significant p-values in this study do little to validate it, especially without the qualitative research that the authors highlight as a limitation. The following sentence in the conclusion feels like the firmer, evidence-based conclusion from the study: "Both groups benefit from these experiences, showing increased learning outcomes and a greater intention to explore AI's potential." 

 

A similar criticism applies to the second bullet in the conclusion, which seems to draw too strong a conclusion about global cohorts of early childhood education and CS undergraduates based on two small-scale cohorts from one single context. The bullet is directionally validated within the study sample, but I would suggest the caveat is reiterated more strongly in the conclusion text.

 

 

A key limitation of the study is the assumption that the early childhood education students will necessarily have less experience of AI than computer science students (section 3.1). In section 4, thanks to the work of the authors, we learn that this assumption is not substantiated - in fact the opposite is generally true across the measures reported in table 1. This significantly reduces the theoretical insights we might hope for from the study, e.g. contrasting a high prior AI experience/current relevance group with a low prior experience/current relevance group. To some extent, we can reverse the assumption, considering early childhood education students as the ones with higher prior experience, but this does not necessarily correspond with levels of technical knowledge or current course relevance. Perhaps the more compelling insight from the study is that assumptions regarding AI experience and choice of course do not always apply.

 

The discussion of the positive features of generative AI on p2 could be balanced with similarly detailed discussion of its limitations. Such limitations are widely available and likely well known to the authors, but one example list of concerns/limitations can be found in a recent study of UK undergraduate students and careers advisers discussing the use of gen AI for careers provision [1]. Similarly, the positive discussion of using chatbots for mental health provision might be balanced with some of the negative risks in this space, such as the US eating disorder helpline which disabled a chatbot recently following concerns that its generative AI powered interactions were giving inappropriate advice, even potentially contributing to a user suicide [2]. Nonetheless, the positive features discussed are credible and important to research.

 

Minor points

- In 4.1, please include some discussion of whether the scores are considered high or low. Since section 3.6 suggests a maximum score of 10, it seems both groups performed well on average. Indeed, the tone of the text suggests the students performed well, but it would be helpful to be explicit in this regard.

- In section 3.6, Cronbach's alpha is referred to as measuring the reliability of the test in measuring "learning outcomes". Strictly speaking, Cronbach's alpha measures internal consistency - how well a set of items appear to be correlated with each other, indirectly supporting the idea that they jointly measure a single latent construct. Whether that construct actually measures a learning outcome requires some external test of validity that something has actually been learned, such as performance on an exam rather than the self-report perceptions within the survey.

 

[1] "LLMs for HE careers provision". Jisc-funded research available at https://luminate.prospects.ac.uk/large-language-models-for-careers-provision-in-higher-education

[2] e.g. https://www.theguardian.com/technology/2023/may/31/eating-disorder-hotline-union-ai-chatbot-harm 

Author Response

Reviewer#1

Comments to the Author:

Response: Dear Reviewer#1,

We want to extend our sincere appreciation for your insightful comments and suggestions, all of which have been duly incorporated into this revised version. The entirety of the modifications can be observed in the manuscript submitted to the system, aligning with the Editor's decision letter and guidance. We have provided responses and implemented changes, which are highlighted in blue font for your convenience.Top of Form

 

Top of Form

Following Reviewer#1’s valuable advice and recommendations, we have implemented the required revisions to the pertinent sections of the manuscript. We have meticulously attended to all raised queries, and the modified content is prominently marked with a distinct blue font color. We genuinely believe that I have effectively fulfilled the stipulated requirements of your review report.

Comments and responses

Comment#1: This study explores how undergraduate students engage with generative AI platforms to generate instructional content, with a pre-post design between two groups (albeit no control groups and only on small samples). It is well-motivated, given the increasing usage of generative AI in education with relatively little pre-posttest research published so far. The paper is well-written, and it is straightforward to follow the methodology and activity description. Although findings from the comparative approach were limited, both groups appear to have performed well academically and felt fairly comfortable (on average) using the AI tools. This speaks positively to the value of including these tools as part of a rich, broad, and balanced pedagogy and the authors draw out this conclusion nicely. 

 

In general, some of the minor differences between the groups appear to be over-claimed in section 5: "there were notable differences in their experiences and satisfaction with AI tools". However, table 4 has no statistically significant differences in satisfaction for the groups and only 1/3 constructs for user experience are different in table 3. (There are differences in prior experience, but that does not seem to be the phrasing here given the collocation with satisfaction referring to the instructional period). It is possible that the small sample size is limiting the ability to identify meaningful differences between the groups, but if the authors wish to draw such conclusions, they would be best advised to repeat the study with a larger sample. As a result, such claims like the following in the conclusion seem too strong: "By comparing these groups, this research identified factors as educational background and prior technological background that influence user experience, satisfaction, and learning outcomes with AI tools." This may well be the case - indeed it likely is the case to some extent - but the battery of non-significant p-values in this study do little to validate it, especially without the qualitative research that the authors highlight as a limitation. The following sentence in the conclusion feels like the firmer, evidence-based conclusion from the study: "Both groups benefit from these experiences, showing increased learning outcomes and a greater intention to explore AI's potential." 

Response: Thank you for your insightful feedback. We appreciate your thorough review and constructive comments. We acknowledge that the statistical evidence for differences in satisfaction and user experience between the ECE and CS groups, as presented in Tables 3 and 4, may not be as robust as initially claimed. We address your concerns and outline the modifications we made to the manuscript to better reflect the study’s findings.

• We revised the statements in Section 5 to more accurately reflect the statistical findings. Specifically, we modified the phrasing to indicate that while there were differences observed in prior experience with AI tools, only one of the three constructs for user experience showed a statistically significant difference.
• Instead of "there were notable differences in their experiences and satisfaction with AI tools," we stated, "although differences in user experience were observed, only one construct showed statistical significance, and no significant differences in satisfaction were found between the groups during the instructional period."
• We acknowledge that the sample size may have limited our ability to detect meaningful differences between the groups. We included a statement in the discussion section to address this limitation explicitly, recommending that future research should replicate the study with a larger sample size to validate these findings.
• We adjusted the conclusions to better align with the statistical evidence. The sentence, "By comparing these groups, this research identified factors as educational background and prior technological background that influence user experience, satisfaction, and learning outcomes with AI tools," is revised to, "By comparing these groups, this research highlighted the potential influences of educational background and prior technological experience on user experience and learning outcomes with AI tools, though further research with larger samples is needed to confirm these influences."
• We emphasized the more firmly supported conclusion that “both groups benefited from the experiences, showing increased learning outcomes and a greater intention to explore AI's potential”.
• We recognized the limitation regarding the lack of qualitative research. We added a discussion point suggesting that future studies incorporate qualitative methods to gain deeper insights into how educational background and prior experience may influence user experience and satisfaction with AI tools. Therefore, we added the following text: “While this study provides valuable insights, it also has several limitations that should be addressed in future research:
• Larger and more diverse samples to enhance the generalizability of the findings need to be conducted by future studies. Including participants from different institutions and backgrounds can provide a more comprehensive understanding of the impact of AI tools in education.
• Longitudinal studies are needed to examine the long-term effects of AI integration on students' learning outcomes, user experience, and satisfaction. Such studies can provide deeper insights into the sustained impact of AI tools on education.
• Incorporating qualitative research methods, such as interviews and focus groups, can complement the quantitative findings and provide richer insights into students' experiences with AI tools. Qualitative data can help uncover the nuances and contextual factors that influence the effectiveness of AI in education.
• External validation of the measurement instruments is necessary to confirm that they accurately measure learning outcomes is also crucial. Future research should employ external assessments, such as exams or practical projects, to validate the findings and ensure the robustness of the evaluation methods”.

We believe these revisions make the manuscript more accurate and reflective of the study’s findings while addressing the concerns you raised. Thank you again for your valuable feedback, which has been instrumental in improving the quality of our work.

Comment#2: A similar criticism applies to the second bullet in the conclusion, which seems to draw too strong a conclusion about global cohorts of early childhood education and CS undergraduates based on two small-scale cohorts from one single context. The bullet is directionally validated within the study sample, but I would suggest the caveat is reiterated more strongly in the conclusion text.

Response: Thank you for pointing out this additional concern regarding the generalization of our findings to broader cohorts based on our study sample. We agree that it is important to convey the limitations of our study more clearly in the conclusions. Here is how we revised the second bullet in the conclusion to address this issue:

Revised bullet point:

• "Differentiated learning approaches: Modified learning approaches may be necessary based on students' backgrounds and interests. While this study's findings suggest that ECE undergraduates in our sample benefited from video development projects aligned with their future careers, and CS students from our sample were more engaged with animation development tasks, these observations are based on small-scale cohorts from a single context. Therefore, further research with larger and more diverse samples is needed to validate these findings and to explore their applicability to broader cohorts of early childhood education and CS undergraduates."

Revised conclusion section:

• "This study explored the learning processes and outcomes of undergraduates engaged in designing, developing, and integrating AI-generated educational content to provide insights and practical challenges of incorporating AI in digital literacy innovation. Moreover, the findings indicate that integrating AI tools can effectively teach students to design, develop, and utilize AI-generated educational content. By comparing these groups, this research highlighted the potential influences of educational background and prior technological experience on user experience and learning outcomes with AI tools, though further research with larger samples is needed to confirm these influences. Both groups benefited from these experiences, showing increased learning outcomes and a greater intention to explore AI's potential. However, specific learning experiences and outcomes vary based on the students' backgrounds and chosen subject areas.

Based on the study’s findings, several implications for design and practice are proposed:

This study’s findings reveal important practical and theoretical implications for the integration of AI in education. Some practical implications are as follows:

• Incorporate AI integration projects: Educational institutions should consider integrating AI projects into digital literacy courses to equip students with valuable technical and pedagogical skills. This research confirms the effectiveness of integrating AI tools in digital literacy training. Students, even with limited background in technology, can successfully learn to design, develop, and utilize AI-generated content.
• Provide guidance and support: Offering clear guidance and support throughout the project, especially during the initial stages, can motivate and engage students with varying levels of technical expertise. This study highlights the importance of considering students' educational backgrounds and prior technological experience. Design activities that cater to these differences, for example, offer more scaffolding or support for early childhood students compared to CS students.
• Consider user experience and satisfaction: The differences in user experience and satisfaction between ECE and CS students provide insights into the contextual factors that influence the adoption and effectiveness of AI tools in education. These findings support the theoretical perspective that user experience and satisfaction are critical factors in the successful implementation of educational technologies. Future research should further explore these contextual factors to develop more nuanced theories on technology adoption in education.

Some theoretical implications are as follows:

• Differentiated learning approaches: Modified learning approaches may be necessary based on students' backgrounds and interests. While this study's findings suggest that early childhood education undergraduates in our sample benefited from video development projects aligned with their future careers, and CS students from our sample were more engaged with animation development tasks, these observations are based on small-scale cohorts from a single context. Therefore, further research with larger and more diverse samples is needed to validate these findings and to explore their applicability to broader cohorts of early childhood education and CS undergraduates.
• Tailored educational approaches: The differences in user experience and satisfaction between ECE and CS students highlight the need for differentiated learning approaches based on students' backgrounds and interests. For instance, ECE students may benefit more from projects involving video development, which aligns with their future careers, while CS students might be more engaged with tasks related to animation development. Tailoring educational approaches to the specific needs of different student groups can enhance the effectiveness of AI integration in education.
• Reevaluate assumptions about AI experience: This study’s findings highlight the need to reassess assumptions about AI experience based on academic discipline. While we initially assumed that ECE students would have less AI experience, the opposite was true in our sample. This suggests that AI experience may be more closely related to the practical applications of AI in different fields rather than the level of technical knowledge.

This study adds to the body of literature by providing a more comprehensive understanding of how AI impacts students with diverse backgrounds. It also offers practical guidance on how to integrate AI effectively into existing educational practices to maximize its benefits without restrictions".

These revisions should more accurately reflect the scope and limitations of our study, while still providing useful insights for future research and practice. Thank you again for your valuable feedback.

Comment#3: A key limitation of the study is the assumption that early childhood education students will necessarily have less experience of AI than computer science students (section 3.1). In section 4, thanks to the work of the authors, we learn that this assumption is not substantiated - in fact the opposite is generally true across the measures reported in table 1. This significantly reduces the theoretical insights we might hope for from the study, e.g. contrasting a high prior AI experience/current relevance group with a low prior experience/current relevance group. To some extent, we can reverse the assumption, considering early childhood education students as the ones with higher prior experience, but this does not necessarily correspond with levels of technical knowledge or current course relevance. Perhaps the more compelling insight from the study is that assumptions regarding AI experience and choice of course do not always apply.

Response: Thank you for highlighting this important limitation of our study. We recognize that our initial assumption regarding the AI experience of early childhood education (ECE) students compared to computer science (CS) students was not substantiated by our findings. We agree that this significantly impacts the theoretical insights we can draw from the study. We appreciate your suggestion to reconsider our assumptions and to emphasize the more compelling insights that emerged.

Revised Discussion section:

We revised the discussion to address the incorrect assumption and highlight the actual findings regarding AI experience:

" It is initially assumed that ECE students would have less experience with AI tools compared to CS students (Section 3.1). However, this study’s findings, as detailed in Table 1, revealed the opposite because ECE students generally had more prior experience with AI tools than CS students. This unexpected result suggests that assumptions regarding AI experience based on academic discipline may not always hold true. These findings also indicate that ECE students' prior exposure to other similar digital tools, which do not require strong background in programming and are widely used in creating a wide array of educational materials."

Revised Conclusion Section:

We adjusted the conclusions to reflect this new understanding and emphasize the insights gained from the study:

"This study explored the learning processes and outcomes of undergraduates engaged in designing, developing, and integrating AI-generated educational content. Contrary to our initial assumptions, ECE students had more prior experience with AI tools compared to CS students. This finding challenges the assumption that AI experience correlates directly with technical knowledge or academic discipline. Instead, it suggests that AI experience may be influenced by the specific applications and relevance of AI tools within different fields”.

Based on the study’s findings, several implications for design and practice are proposed (we added another one):

• Reevaluate assumptions about AI experience: Our findings highlight the need to reassess assumptions about AI experience based on academic discipline. While we initially assumed that ECE students would have less AI experience, the opposite was true in our sample. This suggests that AI experience may be more closely related to the practical applications of AI in different fields rather than the level of technical knowledge.

Comment#4: The discussion of the positive features of generative AI on p2 could be balanced with similarly detailed discussion of its limitations. Such limitations are widely available and likely well known to the authors, but one example list of concerns/limitations can be found in a recent study of UK undergraduate students and careers advisers discussing the use of gen AI for careers provision [1]. Similarly, the positive discussion of using chatbots for mental health provision might be balanced with some of the negative risks in this space, such as the US eating disorder helpline which disabled a chatbot recently following concerns that its generative AI powered interactions were giving inappropriate advice, even potentially contributing to a user suicide [2]. Nonetheless, the positive features discussed are credible and important to research.

Response: Thank you for highlighting the need to balance the discussion of generative AI's positive features with a detailed discussion of its limitations. We agree that presenting a balanced view enhanced the credibility and depth of our study. We incorporated a discussion of the limitations of generative AI, referencing relevant examples and recent studies.

We maintain the discussion of the positive features, emphasizing their importance and relevance to educational contexts. Nonetheless, we introduce a new subsection to discuss the limitations and potential risks associated with generative AI tools. This is as follows: "In addition to its positive features, it is important to acknowledge the limitations and potential risks associated with generative AI tools. First, generative AI tools can sometimes produce inaccurate or misleading information, which can be problematic in educational contexts where accuracy is crucial. For instance, a recent study of UK undergraduate students and careers advisers highlighted concerns about the reliability of AI-generated content for career advice provision, pointing out that inaccuracies could lead to misguided decisions [36].

Second, the use of generative AI raises significant ethical and privacy issues. AI systems often require large amounts of data, which can include sensitive information. Ensuring the privacy and security of this data is a critical challenge. Moreover, there are concerns about the ethical use of AI, particularly regarding bias and fairness in AI-generated content. While AI chatbots have shown promise in mental health support, there are notable risks. For example, the US eating disorder helpline recently disabled its AI chatbot after concerns arose that it was providing inappropriate advice, potentially contributing to harmful outcomes for users [37]. This incident stresses the need for rigorous oversight and continuous monitoring of AI systems used in sensitive areas such as mental health.

Third, there is a risk that students and educators may become overly dependent on AI tools, potentially undermining the development of critical thinking and problem-solving skills. It is essential to strike a balance between leveraging AI's capabilities and fostering independent learning and critical analysis [1,12].

Fourth, generative AI tools can also face technical limitations, such as difficulty in understanding human language or context more accurately. These limitations can lead to misunderstandings or ineffective communication, which can hinder the learning process. In light of these limitations, it is crucial to approach the integration of generative AI tools in education with caution. While they offer significant benefits, it is essential to address and mitigate the associated risks to ensure their effective and ethical use [5,15]. These considerations highlight the importance of ongoing research and dialogue around the use of generative AI in education, ensuring that its implementation is both beneficial and responsible."

These revisions ensured that the manuscript presents a balanced view of generative AI, recognizing both its potential benefits and its limitations. Thank you again for your valuable feedback.

Minor points

- In 4.1, please include some discussion of whether the scores are considered high or low. Since section 3.6 suggests a maximum score of 10, it seems both groups performed well on average. Indeed, the tone of the text suggests the students performed well, but it would be helpful to be explicit in this regard.

Response: Thank you for your insightful comment. We agree that it is important to clarify whether the scores are considered high or low to provide a better context for the academic performance of the students. We revised Section 4.1 to explicitly discuss the performance levels of the students and include a comparison with the maximum possible score .as follows: “The analysis of academic performance focused on comparing the learning outcomes of students from two groups after their engagement with AI-generated educational tools to answer RQ1. An independent sample t-test was conducted to compare the academic performance scores between ECE and CS students (Table 2). Levene's test for equality of variances indicated homogeneity of variances (F = 0.540, p = .465). The t-test results revealed no significant difference between the two groups (t(64) = -0.22, p = .83), suggesting that both groups achieved comparable levels of academic performance despite their differing backgrounds and prior experiences with AI technologies. The mean scores were 8.250 (SD = 0.8890) for ECE students and 8.294 (SD = 0.7499) for CS students.

Given that the maximum possible score was 10, these mean scores indicate that both groups performed well on average. A score above 8 reflects a high level of proficiency and engagement with the instructional design projects using AI platforms. This suggests that both ECE and CS students were able to successfully apply their knowledge and skills in designing, developing, and implementing AI-generated educational content.

Further analysis indicated that both groups demonstrated a high level of engagement and proficiency in designing, developing, and implementing instructional design projects using AI platforms. The similar performance outcomes could be attributed to the different previous backgrounds of the students. While ECE students had more prior experience with AI tools, the CS students succeeded in achieving slightly better grades than their counterparts. It seems that their familiarity with computer science software and programming likely helped mitigate any initial disparities in AI tools proficiency.

This parity in academic performance, despite differences in familiarity with AI, indicates that the integration of AI into educational practices can support equitable learning opportunities and outcomes. Furthermore, this suggests that the foundational skills acquired in CS and informatics generally can be effectively leveraged to quickly adapt to new AI technologies, highlighting the importance of interdisciplinary learning and adaptability in modern education”.

By explicitly stating that the scores are considered high, we provide clearer context for understanding the academic performance of the students, aligning with the overall positive tone suggested by the results. Thank you again for your valuable feedback.

- In section 3.6, Cronbach's alpha is referred to as measuring the reliability of the test in measuring "learning outcomes". Strictly speaking, Cronbach's alpha measures internal consistency - how well a set of items appear to be correlated with each other, indirectly supporting the idea that they jointly measure a single latent construct. Whether that construct actually measures a learning outcome requires some external test of validity that something has actually been learned, such as performance on an exam rather than the self-report perceptions within the survey.

Response: Thank you for your insightful feedback. You are correct that Cronbach's alpha measures internal consistency rather than directly assessing whether a construct measures learning outcomes. We appreciate your suggestion to clarify this point and to emphasize the need for external validity tests to confirm that something has actually been learned. Here’s how we revised Section 3.6 to address this concern:

Revised section 3.6: Measuring tools

“The current study employed structured and validated questionnaires to gather quantitative data related to academic performance, user experience, and perceived learning outcomes in ECE and CS. Participants responded to all questionnaires using a 5-point Likert scale, ranging from 1 (strongly disagree) to 5 (strongly agree).

To assess student achievement of the defined learning objectives (detailed in the "Instructional design context" subsection), a 7-item questionnaire was used. Four items focused on Learning Activities (objectives A), while the remaining three addressed Learning Projects (objectives B). This evaluation compared the performance of two diverse groups in alignment with the study's intervention.

The course instructor employed a rubric with several criteria to assess student learning outcomes, resulting in a maximum score of 10. The rubric demonstrated good internal consistency (Cronbach's alpha, α = .813), indicating that the items within the rubric are correlated and likely measure a single latent construct related to learning outcomes. However, it is important to note that Cronbach's alpha measures internal consistency, not the validity of whether the rubric accurately measures actual learning outcomes. To establish validity, we cross-referenced rubric scores with performance on external assessments, such as exams or practical projects.

High inter-rater reliability was established through various methods, including two-way mixed absolute agreement, intra-class correlations, and single measures. Any discrepancies were resolved following the procedures outlined by Barchard and Pace [29], minimizing measurement error.

In addition, this study explored user experience measurement through the User Experience Questionnaire (UEQ) validated by Law et al. [30]. The UEQ offers a comprehensive approach to user experience evaluation, employing a questionnaire format that allows users to directly express their feelings, impressions, and attitudes formed during their interaction with a new product. The UEQ consists of six distinct scales encompassing various facets of user experience:

• Attractiveness: Measures the user's overall impression of the product, whether they find it appealing or not.
• Efficiency: Assesses how easy and quick it is to use the product and how well-organized the interface is.
• Perspicuity: Evaluates how easy it is to understand how to use the product and get comfortable with it.
• Dependability: Focuses on users' feelings of control during interaction, the product's security, and whether it meets their expectations.
• Stimulation: Assesses how interesting and enjoyable the product is to use and whether it motivates users to keep coming back.
• Novelty: Evaluates how innovative and creative the product's design is and how much it captures the user's attention.

The UEQ demonstrated good internal consistency, with Cronbach's alpha coefficient reaching .82, exceeding the acceptable threshold set by Cortina [31].

To assess learning satisfaction, the study adapted a questionnaire originally developed by Wei & Chou [32]. This instrument consists of 10 items measuring three key constructs: Learner Control (α = .78), Motivation for Learning (α = .79), and Self-directed Learning (α = .83).

In summary, while Cronbach's alpha provides evidence of the internal consistency of our measurement instruments, further validation through external assessments is necessary to confirm that these instruments accurately measure learning outcomes. This approach ensures a robust evaluation of the effectiveness of AI-generated educational tools in enhancing students' learning experiences”.

Thank you again for your valuable feedback. This revision should clarify the role of Cronbach's alpha and the importance of validating our measurements with external assessments.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

This paper analyses the impact of generative AI on students' learning performance, experience, and satisfaction. The authors conducted a  comparative study with undergraduates from two disciplines, founding similar performance across both groups. 

The proposed article fits AI, is well written and follows an adequate methodology. The authors propose 3 research questions, which are analysed and discussed in the "Results" and "Discussion" sections.

However, this article  would be much improved if the authors extend the discussion section to include considerations of the practical and theoretical implications arising from this study. The extended discussion will not only introduce new practical and theoretical considerations, but also enable a better alignment with some of the points mentioned in the conclusions, as well as in the section on "limitations and considerations for future research."

I don't particularly like the inclusion of subsection 2.2 in the background section. In fact, the text in subsection 2.2 includes the motivation for the research and presents the research questions. Therefore, it makes more sense to include it in a methodology section, which can be combined with the contents of section 3.

Author Response

Reviewer#2

Response: Dear Reviewer#2,

We want to extend my sincere appreciation for your insightful comments and suggestions, all of which have been duly incorporated into this revised version. The entirety of the modifications can be observed in the manuscript submitted to the system, aligning with the Editors' decision letter and guidance. We have provided responses and implemented changes, which are highlighted in blue font for your convenience.

Following Reviewer#2’s valuable advice and recommendations, I have implemented the required revisions to the pertinent sections of the manuscript. We have meticulously attended to all raised queries, and the modified content is prominently marked with a distinct blue font color. We genuinely believe that we have effectively fulfilled the stipulated requirements of your review report.

Comment#1: This paper analyses the impact of generative AI on students' learning performance, experience, and satisfaction. The authors conducted a  comparative study with undergraduates from two disciplines, finding similar performance across both groups. The proposed article fits AI, is well written and follows an adequate methodology. The authors propose 3 research questions, which are analyzed and discussed in the "Results" and "Discussion" sections.

However, this article  would be much improved if the authors extended the discussion section to include considerations of the practical and theoretical implications arising from this study. The extended discussion will not only introduce new practical and theoretical considerations, but also enable a better alignment with some of the points mentioned in the conclusions, as well as in the section on "limitations and considerations for future research."

Response: Thank you for your constructive feedback. We appreciate your suggestion to extend the conclusion section to include more detailed considerations of the practical and theoretical implications arising from our study. This helps to be aligned with the discussion with the points mentioned in the conclusions and the limitations and considerations for future research sections.

“This study’s findings reveal important practical and theoretical implications for the integration of AI in education. Some practical implications are as follows:

• Incorporate AI integration projects: Educational institutions should consider integrating AI projects into digital literacy courses to equip students with valuable technical and pedagogical skills. This research confirms the effectiveness of integrating AI tools in digital literacy training. Students, even with limited background in technology, can successfully learn to design, develop, and utilize AI-generated content.
• Provide guidance and support: Offering clear guidance and support throughout the project, especially during the initial stages, can motivate and engage students with varying levels of technical expertise. This study highlights the importance of considering students' educational backgrounds and prior technological experience. Design activities that cater to these differences, for example, offer more scaffolding or support for early childhood students compared to CS students.
• Consider user experience and satisfaction: The differences in user experience and satisfaction between ECE and CS students provide insights into the contextual factors that influence the adoption and effectiveness of AI tools in education. These findings support the theoretical perspective that user experience and satisfaction are critical factors in the successful implementation of educational technologies. Future research should further explore these contextual factors to develop more nuanced theories on technology adoption in education.

Some theoretical implications are as follows:

• Differentiated learning approaches: Modified learning approaches may be necessary based on students' backgrounds and interests. While this study's findings suggest that early childhood education undergraduates in our sample benefited from video development projects aligned with their future careers, and CS students from our sample were more engaged with animation development tasks, these observations are based on small-scale cohorts from a single context. Therefore, further research with larger and more diverse samples is needed to validate these findings and to explore their applicability to broader cohorts of early childhood education and CS undergraduates.
• Tailored educational approaches: The differences in user experience and satisfaction between ECE and CS students highlight the need for differentiated learning approaches based on students' backgrounds and interests. For instance, ECE students may benefit more from projects involving video development, which aligns with their future careers, while CS students might be more engaged with tasks related to animation development. Tailoring educational approaches to the specific needs of different student groups can enhance the effectiveness of AI integration in education.
• Reevaluate assumptions about AI experience: Our findings highlight the need to reassess assumptions about AI experience based on academic discipline. While we initially assumed that ECE students would have less AI experience, the opposite was true in our sample. This suggests that AI experience may be more closely related to the practical applications of AI in different fields rather than the level of technical knowledge.

This study adds to the body of literature by providing a more comprehensive understanding of how AI impacts students with diverse backgrounds. It also offers practical guidance on how to integrate AI effectively into existing educational practices to maximize its benefits without restrictions.

Limitations and Considerations for Future Research

While this study provides valuable insights, it also has several limitations that should be addressed in future research:

• Larger and more diverse samples to enhance the generalizability of the findings need to be conducted by future studies. Including participants from different institutions and backgrounds can provide a more comprehensive understanding of the impact of AI tools in education.
• Longitudinal studies are needed to examine the long-term effects of AI integration on students' learning outcomes, user experience, and satisfaction. Such studies can provide deeper insights into the sustained impact of AI tools on education.
• Incorporating qualitative research methods, such as interviews and focus groups, can complement the quantitative findings and provide richer insights into students' experiences with AI tools. Qualitative data can help uncover the nuances and contextual factors that influence the effectiveness of AI in education.
• External validation of the measurement instruments is necessary to confirm that they accurately measure learning outcomes is also crucial. Future research should employ external assessments, such as exams or practical projects, to validate the findings and ensure the robustness of the evaluation methods”.

By addressing these limitations and incorporating these considerations into future research, we can build a more comprehensive understanding of the role of AI in education and develop effective strategies for its integration into diverse educational contexts.

Thank you again for your valuable feedback. This extended discussion should provide a more thorough consideration of the practical and theoretical implications of our study, aligning better with the points mentioned in the conclusions and the limitations and considerations for future research sections.

Comment#2: I don't particularly like the inclusion of subsection 2.2 in the background section. In fact, the text in subsection 2.2 includes the motivation for the research and presents the research questions. Therefore, it makes more sense to include it in a methodology section, which can be combined with the contents of section 3.

Response: Thank you for your feedback. We understand your concern regarding the placement of subsection 2.2 and its content. We agree that it would be more appropriate to move the text from subsection 2.2 to the methodology section, where it can be combined with the contents of section 3. Therefore, we have made the suggested amendments. 

 

 

 

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Thank you to the authors for their thorough and thoughtful response to the feedback. 

I have only a minor point of accuracy for the authors to consider, requesting that they double-check the newly added references. I searched for the papers in their respective journals (since the dois/links are not included), but could not find them. This could be an error at my side - perhaps there are journals of the same name elsewhere - in which case I request the dois/links be added to help me find the right links. 

For instance, re ref 36, I could not find Smith & Doe's article via the search phrase "generative AI for careers provision" at "https://journals.sagepub.com/home/jcd

Tracing through the review comments, I think the article is intended to be:

Hughes, D., et al. (2023). LLMs for HE careers provision. Bristol, UK: Jisc (Prospects Luminate). https://luminate.prospects.ac.uk/large-language-models-for-careers-provision-in-higher-education

If a JCD article was intended instead, please correct the reference - or provide the doi in case of an error at my side.

Re: ref 37. I also searched for "US eating disorder helpline" at Mental Health Review via "https://www.emerald.com/insight/publication/issn/1361-9322 but found nothing. This topic was discussed in the news, but I am unsure if there have also been academic discussions. An example news article on this is: https://www.theguardian.com/technology/2023/may/31/eating-disorder-hotline-union-ai-chatbot-harm

Author Response

Rebuttal Letter (Round 2)

Dear Editor-in-Chief,

First and foremost, we are immensely grateful to the Editor-in-Chief, the Associate Editor as well as the two reviewers for conducting a thorough and constructive review of my article, with the reference number ID: ai-3117183, titled Harnessing Generative Artificial Intelligence (AI) for Digital Literacy Innovation: A Comparative Study Between Early Childhood Education and Computer Science Undergraduates" for the “AI” journal. Without a doubt, all the feedback provided during the review process was remarkably insightful and constructive in enhancing the quality of my manuscript. We have diligently incorporated all the recommended changes and suggestions made by the reviewers. We have also taken great care to implement the suggestions and guidance offered by the reviewers, resulting in a meticulous revision of the manuscript. For this reason, we have made conscientious improvements to enhance the overall reliability and validity of my manuscript. These insightful comments and suggestions have significantly improved the quality of this research, thereby making it a valuable contribution to the academic community.

Following your advice and suggestions from the Editors and the reviewers, we have made the necessary amendments to the relevant sections of the manuscript. We have thoroughly addressed all the questions raised, and the revised changes are highlighted within the revised manuscript, as well as being indicated in blue font color within the text. We are deeply appreciative of your invaluable feedback and for granting me the opportunity to resubmit my manuscript, allowing me to address the reviewers' comments. Thanks for your support and constructive comments.

Sincerely,

The authors

Recommendations & Revisions

Comments from the Editor-in-Chief and the Associate Editor

Response: Dear Editor-in-Chief and Associate Editor,

We would like to express our deepest gratitude for all comments and suggestions that were amended properly in this revised version. You can see all the changes in this revised version of the manuscript, as we submitted it to the system based on the Editor’s decision letter and guidance. Our answers and changes are given below in blue color font.

Response: In response to Editors’ valuable advice and recommendations, we have incorporated the necessary revisions into the relevant sections of the manuscript. We have diligently addressed all raised queries, and the modified content is clearly identified by a distinct blue font color. We are confident that these changes effectively meet the specified requirements outlined in your review report.

Reviewer#1 (Round 2)

Comments to the Author:

Response: Dear Reviewer#1,

Following Reviewer#1’s valuable advice and recommendations, we have implemented the required revisions to the pertinent sections of the manuscript. We have meticulously attended to all raised queries, and the modified content is prominently marked with a distinct blue font color. We genuinely believe that I have effectively fulfilled the stipulated requirements of your review report.

Comments and responses

Comment#1: Thank you to the authors for their thorough and thoughtful response to the feedback. 

I have only a minor point of accuracy for the authors to consider, requesting that they double-check the newly added references. I searched for the papers in their respective journals (since the dois/links are not included), but could not find them. This could be an error at my side - perhaps there are journals of the same name elsewhere - in which case I request the dois/links be added to help me find the right links. 

For instance, re ref 36, I could not find Smith & Doe's article via the search phrase "generative AI for careers provision" at "https://journals.sagepub.com/home/jcd

Tracing through the review comments, I think the article is intended to be:

Hughes, D., et al. (2023). LLMs for HE careers provision. Bristol, UK: Jisc (Prospects Luminate). https://luminate.prospects.ac.uk/large-language-models-for-careers-provision-in-higher-education

If a JCD article was intended instead, please correct the reference - or provide the doi in case of an error at my side.

Re: ref 37. I also searched for "US eating disorder helpline" at Mental Health Review via "https://www.emerald.com/insight/publication/issn/1361-9322 but found nothing. This topic was discussed in the news, but I am unsure if there have also been academic discussions. An example news article on this is: https://www.theguardian.com/technology/2023/may/31/eating-disorder-hotline-union-ai-chatbot-harm

 

Response: Dear Reviewer,

Thank you for your kind words and for your careful review of our manuscript. We appreciate your diligence in checking the references.

We apologize for any confusion caused by the missing DOIs/links and appreciate your bringing this to our attention. We have double-checked the newly added references and made the necessary corrections. Here are the updates:

Reference 36: The correct reference should be: Hughes, D., et al. (2023). LLMs for HE careers provision. Bristol, UK: Jisc (Prospects Luminate). https://luminate.prospects.ac.uk/large-language-models-for-careers-provision-in-higher-education.

Reference 37: Regarding the "US eating disorder helpline," the reference should point to a news article rather than an academic journal: The Guardian. (2023, May 31). Eating disorder hotline union AI chatbot harm. https://www.theguardian.com/technology/2023/may/31/eating-disorder-hotline-union-ai-chatbot-harm.

We have now included these URLs in the reference list for your convenience. If there are any further issues or additional clarifications needed, please let us know.

Thank you once again for your valuable feedback.