Re-Evaluating Components of Classical Educational Theories in AI-Enhanced Learning: An Empirical Study on Student Engagement

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The purpose of this paper is to identify and validate the factors that influence student participation in a learning environment where artificial intelligence is integrated into the curriculum. The integration of artificial intelligence offers learning experiences that deviate from the traditional and require targeted efforts.

The paper is well-documented based on Bandura's Social Cognitive Theory and Self-Determination Theory. The former is appropriate as it provides a framework for understanding how individuals learn in their social environment and the environmental factors that influence behavior. The latter relates to the concepts of autonomy, competence, and relatedness, which, when increased, also elevate motivation and engagement. Additionally, the study draws on other relevant concepts such as motivation, personalized learning, self-regulation technology, and artificial intelligence.

The four research questions are well-defined and align with the main objective. The survey was administered to a large sample of students, yielding 642 valid responses, which makes the study reliable due to the large number of respondents.

The methodology is explained in detail, including the study context, participants, their characteristics, the medium through which the survey was administered, the data analysis methods, and everything necessary to understand the procedures followed in the study. All of this is consistent with the research objective. The types of analyses used and their suitability for answering the research questions are explained, as well as the process for validating the survey instrument.

Furthermore, the content of the survey is explained in detail and its construction is justified, along with the various factors the survey sought to investigate. The analysis identifies the different questions associated with the components determined for the research. Additionally, the confirmatory factor analysis supports the validity of the factor structures.

From the results, various aspects related to autonomy through the use of artificial intelligence resources and deeper engagement in interaction with this type of intelligence are identified. Self-regulation is also shown to be structured and aligned with the objectives, supported by artificial intelligence.

From this perspective, a novel scale is established to identify student participation in the context of AI-enhanced learning. This will be useful for identifying participation and establishing new mechanisms to enhance learning in these contexts with this new technology, and increasing students' autonomy, self-determination, motivation, and engagement when using this type of technology.

It is a very well-crafted study, well-structured in the paper, and consistent across all its sections. Each section is presented with sufficient depth to understand the theoretical and methodological aspects that underpin the research. The data are interpreted coherently and appropriately in accordance with the research objective. The results are align with the objective and are relevant to the field of knowledge addressed. Regarding the conclusions, they are adequate, but the authors could improve them by opening the discussion to future research on the same topic, the same approach, or what can be achieved in the future in the field of work (lines 620 to 639).

The theoretical references and background are relevant, encompassing recent research without self-citations. The figures and tables presented are clear and rigorously illustrate the data for the reader's complete understanding.

The work is suitable for the scope and purposes of the journal. It is a relevant piece of research in the field of contemporary education, where the door is opened to artificial intelligence and the various ways it can be ethically, honestly, and appropriately used in the classroom to enhance student learning. Undoubtedly, the scale created from this research can become an important and relevant tool for future studies on the topic, thus constituting a significant contribution to the field of knowledge. The use of the English language is comprehensible and appropriate for communicating the knowledge.

Author Response

Response to Reviewer 1's comments:
1. The purpose of this paper is to identify and validate the factors that influence student participation in a learning environment where artificial intelligence is integrated into the curriculum. The integration of artificial intelligence offers learning experiences that deviate from the traditional and require targeted efforts. (Comments not required.)

2. The paper is well-documented based on Bandura's Social Cognitive Theory and Self-Determination Theory. The former is appropriate as it provides a framework for understanding how individuals learn in their social environment and the environmental factors that influence behavior. The latter relates to the concepts of autonomy, competence, and relatedness, which, when increased, also elevate motivation and engagement. Additionally, the study draws on other relevant concepts such as motivation, personalized learning, self-regulation technology, and artificial intelligence. (Thank you for the appreciative words.)

3.The four research questions are well-defined and align with the main objective. The survey was administered to a large sample of students, yielding 642 valid responses, which makes the study reliable due to the large number of respondents. (Thank you for the appreciative words.)

4. The methodology is explained in detail, including the study context, participants, their characteristics, the medium through which the survey was administered, the data analysis methods, and everything necessary to understand the procedures followed in the study. All of this is consistent with the research objective. The types of analyses used and their suitability for answering the research questions are explained, as well as the process for validating the survey instrument. (Thank you for the appreciative words.)

5. Furthermore, the content of the survey is explained in detail and its construction is justified, along with the various factors the survey sought to investigate. The analysis identifies the different questions associated with the components determined for the research. Additionally, the confirmatory factor analysis supports the validity of the factor structures. (Thank you for the appreciative words.)

6. From the results, various aspects related to autonomy through the use of artificial intelligence resources and deeper engagement in interaction with this type of intelligence are identified. Self-regulation is also shown to be structured and aligned with the objectives, supported by artificial intelligence. (Thank you for the appreciative words.)

7. From this perspective, a novel scale is established to identify student participation in the context of AI-enhanced learning. This will be useful for identifying participation and establishing new mechanisms to enhance learning in these contexts with this new technology, and increasing students' autonomy, self-determination, motivation, and engagement when using this type of technology. (Thank you for the appreciative words.)

8. It is a very well-crafted study, well-structured in the paper, and consistent across all its sections. Each section is presented with sufficient depth to understand the theoretical and methodological aspects that underpin the research. The data are interpreted coherently and appropriately in accordance with the research objective. The results are align with the objective and are relevant to the field of knowledge addressed. Regarding the conclusions, they are adequate, but the authors could improve them by opening the discussion to future research on the same topic, the same approach, or what can be achieved in the future in the field of work (lines 620 to 639).

(Thank you for the appreciative words. The Conclusion section has been expanded with the paragraph below.)

Looking to the future, there is a clear need for further research to expand on the findings of this study. One critical avenue for future exploration is conducting longitudinal studies to examine how student engagement with AI tools evolves over extended periods. It may be of particular interest to examine this change from the phase when they used these tools only occasionally in their independent learning, to the phase when they required the use of AI to be consciously integrated into their subjects over a semester. Such studies would provide deeper insights into the long-term impacts of AI integration on educational outcomes. In fact, a longitudinal study has already been conducted as a continuation of this research, and a subsequent paper detailing those findings is currently under review in the same journal. This ongoing research will contribute to a more comprehensive understanding of the sustained effects of AI tools on student engagement and learning success.

Additionally, future research could explore the differential impacts of AI tools across various student demographics, such as age, gender, and field of study, to determine whether certain groups benefit more from AI integration than others. Investigating the role of AI in collaborative learning environments versus individual learning contexts could also yield valuable insights, particularly in understanding how AI tools influence group dynamics and peer learning.

Another promising area for future research is the development and evaluation of AI tools tailored to specific disciplines or learning styles. Customizing AI tools to meet the unique needs of different subjects or adapting them to various learning preferences could further enhance their effectiveness and student engagement.

Furthermore, an exciting direction for future research is the possible use of students' factor scores, derived from the AIELE Scale, as predictors in a machine learning model to forecast students' performance in their studies. By leveraging these factor scores alongside other well-known predictors, educators and institutions could potentially improve the accuracy of performance forecasts, identify students at risk of underperforming, and intervene early with tailored support, thereby improving overall educational outcomes.

In summary, while this study provides a solid foundation for understanding the role of AI in student engagement, the field is ripe for further exploration across these various dimensions.

9. The theoretical references and background are relevant, encompassing recent research without self-citations. The figures and tables presented are clear and rigorously illustrate the data for the reader's complete understanding. (Thank you for the appreciative words.)

10. The work is suitable for the scope and purposes of the journal. It is a relevant piece of research in the field of contemporary education, where the door is opened to artificial intelligence and the various ways it can be ethically, honestly, and appropriately used in the classroom to enhance student learning. Undoubtedly, the scale created from this research can become an important and relevant tool for future studies on the topic, thus constituting a significant contribution to the field of knowledge. The use of the English language is comprehensible and appropriate for communicating the knowledge. (Thank you for the appreciative words.)

Reviewer 2 Report

Comments and Suggestions for Authors

I think that the paper presents a comprehensive analysis of student engagement using AI-based chat tools, using both EFA and CFA to establish construct validity and reliability. The authors successfully identify four distinct components related to student engagement and provide a detailed interpretation of these components within the theoretical frameworks of SCT and SDT.  Excellent paper!

However, there is one area  where it can be improved to enhance its clarity, and contribution to the field of educational technology. 

For the methology, the choice of PCA over other methods should be justified. By addressing this point, the authors can further enhance the clarity and impact of their research.

Author Response

Response to Reviewer 2's comments:

1. I think that the paper presents a comprehensive analysis of student engagement using AI-based chat tools, using both EFA and CFA to establish construct validity and reliability. The authors successfully identify four distinct components related to student engagement and provide a detailed interpretation of these components within the theoretical frameworks of SCT and SDT.  Excellent paper! (Thank you for the appreciative words.)

2. However, there is one area  where it can be improved to enhance its clarity, and contribution to the field of educational technology. For the methology, the choice of PCA over other methods should be justified. By addressing this point, the authors can further enhance the clarity and impact of their research.

(The original manuscript has been expanded with the additional paragraph below.)

The decision to use PCA over other factor extraction methods was based on the fact that PCA provided the highest proportion of the total variance explained in the sample data. We applied several other factor extraction methods, but the factors determined by PCA were most effective in capturing the underlying structure of the data, making it the most suitable choice for our analysis. This allowed us to maximize the explained variance and enhance the robustness of our findings.