The Effectiveness of Educational Robots in Improving Learning Outcomes: A Meta-Analysis
Round 1
Reviewer 1 Report
Comments for author File: 
Comments.pdf
Author Response
Reviewers' Comments and Authors Response
Paper number：sustainability-2240669
Paper title：The effectiveness of educational robots in improving learning outcomes: A meta-analysis
The authors would like to thank the area editor and the reviewers for their precious time and thoughtful comments. We have carefully addressed all the comments. Our responses follow below in boldface text.
Reviewer #1:
1) COMMENT: Literature search and inclusion criteria: Do you think only the web of science and google scholar is enough for searching literature in this domain of research? There is another online database such as Scopus which is considered the largest abstract and citation database of peer-reviewed literature. Also, the following database contains a lot of papers published in the domain of educational robots: ACM Digital Library, Emerald, IEEE Xplore, SAGE Journals, ScienceDirect, SpringerLink, Taylor & Francis Online, MDPI, etc.
The authors should update their search with the Scopus database at least. Also, the date of the search should be included because the database is always changing day by day.
Response：We thank the reviewer for this comment. We agree that there is a need to update the search database and date. As suggested by the reviewer, we have updated the search in Scopus database and re-run the meta-analysis. The date of the search has also been included accordingly.
2) COMMENT: Your second research question is “the effects vary across different situations including educational level, discipline, treatment duration, type of assessment, and robotic type?” Based on what you present in the paper, this question is not answered. The question can be answered based on the following comments: a) Four different level of education was included in this study (i.e., Preschool education, Primary education, Secondary education, and Higher education). It is well known in the literature that the educational level affects students understanding. As such, it will interest the reader if you can include the Correlation between Education Level and the effect of educational robots in improving learning outcomes. b) The treatment duration is an important factor affecting learning outcomes. The effect of the treatment duration on the educational robots learning outcomes should be included. c) Authors report different types of assessments. The impact of assessment on students' learning is an interesting topic in this domain of research. Therefore, comparing the reported type of assessment is an essential part that should be added to the manuscript.
Response：We would like to thank the reviewer for pointing out these questions. We would like to clarify that the purpose of the second research question was to explore whether the four factors, i.e., educational level, subject area, treatment duration and type of assessment, would moderate the effects of educational robots on student learning. To clarify our points, we have rephrased the second research question as follows：
Q2. Does the effect vary by
a) the educational level (pre-school, primary school, secondary school, higher education)?
b) the subject area (social science and humanities, science)?
c) the treatment duration (0-4 weeks, 4-8weeks, above 8weeks)?
d) the type of assessment (exam mark, skill-based measure, attitude)?
e) the robotic type (robotic kits, zoomorphic social robot, humanoid robot)?
Moreover, we add the interaction effects of meta-analysis as a limitation in our revised manuscript as follow:
Finally, although it is central to analyze specific characteristics of the interventions influence the effect of educational robots in formal learning environments, in the future it will be valuable to explore interaction effects among moderators, which can provide valuable information to answer questions such like“do these intervention components amplify or attenuate each other’s effectiveness?”
3) COMMENT: The quality of Figure 2 should be improved
Response：As suggested by the reviewer, we have updated the figure of Risk of bias in the revised manuscript.
4) COMMENT: Please consider changing Figure 3 to a tabular form.
Response：Thanks for the comment. We have edited the content of forest plot in Figure 3 in the revised manuscript.
5) COMMENT: Include the section number in each section of the manuscript.
Response：We follow the suggestion of the reviewer and have added a section number in each section of the manuscript.
6) COMMENT: literature shows that educational robots are a constantly evolving field with the potential to be implemented in education at all levels from kindergarten to university. However, out of more than 70 references cited only 9 references are recent (i.e., 2019 to 2023), with only one reference to 2022 and no 2023 reference. Please consider more recent references.
Response：We have updated the references by including some very recent literature (e.g., Ajlouni, 2023; Hsieh, 2022; Yang et al., 2023) according to our inclusion criteria.
Reviewer 2 Report
The authors have conducted a meta-analysis study to systematically synthesize studies findings on effects of educational robots on students' learning outcomes. The authors have cited 15 peer-reviewed articles that met their selection criteria for their meta-analysis study which included 27 effect sizes with 1165 participants and they also cite several other studies to provide evidence of how educational robots can be employed to improve student academic performance in various courses like English learning, mathematics concepts or engineering topics among others. All references are appropriate and relevant to this research paper.
The English used in this paper is correct and readable. The authors have written their research paper using clear language that can be easily understood by readers of all levels. They also provide definitions for any technical terms they use to ensure clarity throughout the text.
The practical implications of this paper are that it provides researchers and practitioners with insights into what characteristics of educational robots interventions appear to benefit students' learning outcomes, as well as how pedagogical approaches can be applied in various educational settings. This information is useful for guiding the design of future educational robots interventions so they can effectively facilitate student learning. Additionally, findings from moderator analyses provide guidance on which factors should be taken into consideration when designing robotic-based instruction (e.g., course type, education level).
The main disadvantage of this work is that it only focused on the effects of educational robots interventions, and did not consider other factors such as gender differences or socio-economic status which may also influence student learning outcomes. Additionally, since most studies included in the meta-analysis were conducted with children aged 8 to 12 years old, further research should be done to explore how these findings can be applied for different age groups.
I suggest that the authors could further explore other influencing factors related to students' academic performances with educational robots interventions (e.g., gender differences) and discuss potential implications from these findings for future studies/practices involving robotic technology use in education settings.
Author Response
Reviewers’ Comments and Authors Response
Paper number：sustainability-2240669
Paper title：The effectiveness of educational robots in improving learning outcomes: A meta-analysis
The authors would like to thank the area editor and the reviewers for their precious time and thoughtful comments. We have carefully addressed all the comments. Our responses follow below in boldface text.
Reviewer #2:
COMMENT: The main disadvantage of this work is that it only focused on the effects of educational robots interventions, and did not consider other factors such as gender differences or socio-economic status which may also influence student learning outcomes. Additionally, since most studies included in the meta-analysis were conducted with children aged 8 to 12 years old, further research should be done to explore how these findings can be applied for different age groups. I suggest that the authors could further explore other influencing factors related to students' academic performances with educational robots interventions (e.g., gender differences) and discuss potential implications from these findings for future studies/practices involving robotic technology use in education settings.
Response：We appreciate the reviewer for pointing this out. We agree that our research did not take into account all possible influencing factors. In our future research we could address other factors, especially the factors suggested by the reviewer. Therefore, we have added some text in the limitation and future research section at the end of the article to better inform practitioners. In detail:
A first limitation was that this study only focused on several factors that might affect the effects of educational robot interventions. It did not consider other factors such as gender differences or socio-economic status which might also influence student learning outcomes. Future research should consider how these and other factors might be related to students' academic performance involving the use of robotic technology in education settings.
Another limitation was that most randomized-controlled studies included in the meta-analysis were conducted with children aged 8 to 12 years old. Further research should be done to explore how these findings can be applied for different age groups.
Reviewer 3 Report
Dear,
This article aims at presenting an up-to-date and thorough review of the learning effectiveness of educational robots and their surrounding influencing factors (course type, education level, treatment duration, assessment type and robot type), applying statistical and moderator analyses techniques.
The manuscript is well presented, properly written, and statistics have been correctly applied. Even a bias risk analysis is performed (please not that the aesthetics of the risk analysis of Figure 2 is not visually attractive, 0%-25%, etc. figures look a bit blurry). Conclusions are clear and relevant, and are statistically representative. Thus, the article has a great potential for the practitioners of the field.
Technically, I would just suggest going deeper into detail in the justification of the techniques chosen: why sensitivity analysis (quite evident, but necessary), and why moderator analyses (not so evident, so a reference justifying is appropriateness or identity is needed).
Regarding contextualization, the literature review is rather complete, and it even includes 5 fresh references regarding robots and teaching (made the last two years). Nevertheless, I would go a bit further to extend the review, as the abstract itself proposes having and up-to-date and thorough review. I would analyze, thus, the convenience of including these and other fresh references:
TeachBot: Towards teaching robotics fundamentals for human-robot collaboration at work
NS Selby, J Ng, GS Stump, G Westerman, C Traweek… - Heliyon, 2021 - Elsevier
Should robots replace teachers? Mobilisation of AI and learning analytics in education
A Alam - 2021 International Conference on Advances in …, 2021 - ieeexplore.ieee.org
Artificial intelligence-based robots in education: A systematic review of selected SSCI publications
ST Chu, GJ Hwang, YF Tu - Computers and education: Artificial …, 2022 – Elsevier
Concerning style, I just have a minor remark: These sentences (last paragraph of page 11) sound confusing: “However, existing body of research provides limited empirical evidence of student learning gains from educational robots-based instruction (e.g., Alimisis, 2013; Benitti, 2012). Although previous review studies (e.g., Papadopoulos et al., 2020; van Straten et al., 2020)…”: If you read the following lines you realize that we are talking about different things. Nevertheless, being these 2 sentences together, readers can understand that you propose studies made in 2020 are previous to others made in 2013 and 2012. Thus, the proposal is to reformulate them so that conclusions (one of the most important part of your manuscript) are easier to understand.
My suggestion is to go ahead with the approval if these minor changes are tackled. Thank you very much for giving me the opportunity for reviewing this paper.
Best regards,
Reviewer.
Author Response
Reviewers’ Comments and Authors Response
Paper number：sustainability-2240669
Paper title：The effectiveness of educational robots in improving learning outcomes: A meta-analysis
The authors would like to thank the area editor and the reviewers for their precious time and thoughtful comments. We have carefully addressed all the comments. Our responses follow below in boldface text.
Reviewer #3:
1) COMMENT: Technically, I would just suggest going deeper into detail in the justification of the techniques chosen: why sensitivity analysis (quite evident, but necessary), and why moderator analyses (not so evident, so a reference justifying is appropriateness or identity is needed).
Response：We really appreciate the comment of the reviewer. We have added the following sentences in section 2.5 of the revised manuscript to address your concern and hope that this could help to clarify the question about the techniques chosen. In detail:
In meta-analysis, heterogeneity often exists between studies. When multiple moderators are present, they may amplify or attenuate each other's influence on the treatment effectiveness. Hence, moderator analyses were conducted to assess heterogeneity by comparing study subsets (Higgins & Green, 2011; Li et al., 2020).
2) COMMENT: Regarding contextualization, the literature review is rather complete, and it even includes 5 fresh references regarding robots and teaching (made the last two years). Nevertheless, I would go a bit further to extend the review, as the abstract itself proposes having and up-to-date and thorough review. I would analyze, thus, the convenience of including these and other fresh references: TeachBot: Towards teaching robotics fundamentals for human-robot collaboration at work NS Selby, J Ng, GS Stump, G Westerman, C Traweek… - Heliyon, 2021 - Elsevier Should robots replace teachers? Mobilisation of AI and learning analytics in education A Alam - 2021 International Conference on Advances in …, 2021 - ieeexplore.ieee.org Artificial intelligence-based robots in education: A systematic review of selected SSCI publications ST Chu, GJ Hwang, YF Tu - Computers and education: Artificial …, 2022 – Elsevier
Response：Thank you for pointing out this. We have added a couple of very recent literature in the revised manuscript (e.g., Ajlouni,2023; Hsieh, 2022; Yang et al., 2023) according to the inclusion criteria.
3) COMMENT: Concerning style, I just have a minor remark: These sentences (last paragraph of page 11) sound confusing: “However, existing body of research provides limited empirical evidence of student learning gains from educational robots-based instruction (e.g., Alimisis, 2013; Benitti, 2012). Although previous review studies (e.g., Papadopoulos et al., 2020; van Straten et al., 2020)…”: If you read the following lines you realize that we are talking about different things. Nevertheless, being these 2 sentences together, readers can understand that you propose studies made in 2020 are previous to others made in 2013 and 2012. Thus, the proposal is to reformulate them so that conclusions (one of the most important part of your manuscript) are easier to understand.
Response：We thank the reviewer for this comment. We have corrected the sentences in the discussion section of the revised manuscript. In detail:
Despite the proposed advantages of incorporating educational robots in student learning (Alimisis, 2013; Benitti, 2012; Papadopoulos et al., 2020; van Straten et al., 2020), currently researchers have not given particular attention to the overall effectiveness and parameters of successful intervention aiming at the use of educational robots in school settings. To address this gap, this study, therefore, conducted a meta-analysis of the effects of educational robots in the classroom.
Round 2
Reviewer 1 Report
No comment
