A First Approach to Co-Design a Multimodal Pedagogic Conversational Agent with Pre-Service Teachers to Teach Programming in Primary Education
Round 1
Reviewer 1 Report (Previous Reviewer 3)
Comments and Suggestions for AuthorsThe authors have addressed the main points highlighted in the review; as a result, the contribution have significantly improved in terms of readability, contextualization in the literature, description of the study limitations.
However, I still question the overall significance of the experimental part of the work, since pre-service teachers have, at best, a very limited understanding of student needs and/or their experience with PCAs is limited; furthermore, there is no explanation of how the participants to the questionnaire have been recruited (is it a convenience sample? are those student from a specific University course?). On this crucial point, the paper presents very little improvement. For instance, it would be interesting to investigate wheter those pre-service teachers actually implemented PCAs in their teaching activity, or whether their observations match those of in-service teachers.
Furthermore, the authors specify that only 10 out of 44 pre-service teachers provided open-ended suggestions. This part provides very little information and should be discussed synthetically rather than reporting all answers.
Author Response
First of all, we would like to thank the reviewer for the comments given as we believe that have improved the paper very much. The new changes are marked in blue.
The authors have addressed the main points highlighted in the review; as a result, the contribution has significantly improved in terms of readability, contextualization in the literature, description of the study limitations.
Many thanks.
However, I still question the overall significance of the experimental part of the work, since pre-service teachers have, at best, a very limited understanding of student needs and/or their experience with PCAs is limited; furthermore, there is no explanation of how the participants to the questionnaire have been recruited (is it a convenience sample? are those students from a specific University course?). On this crucial point, the paper presents very little improvement. For instance, it would be interesting to investigate whether those pre-service teachers actually implemented PCAs in their teaching activity, or whether their observations match those of in-service teachers.
Sorry, we did not clearly explain that the goal is to recruit pre-service teachers as they are still training and have the possibility of learning PCAs and choose whether to integrate them into their teaching activities. It has now been written on the paper to make it clearer.
As requested by another reviewer, we have added a new Section 2.3 in the literature review to explain why it is so important to train pre-service teachers with new technologies, explore their reactions, and improve their training as they become in-service teachers.
We also agree and believe that comparing the opinions of pre-service and in-service teachers is a very interesting experiment that we have included in our future work.
Furthermore, the authors specify that only 10 out of 44 pre-service teachers provided open-ended suggestions. This part provides very little information and should be discussed synthetically rather than reporting all answers.
We agree. A tag cloud has been added and a synthesis of the answers has been written on the paper instead of just literally reporting all answers.
Reviewer 2 Report (Previous Reviewer 2)
Comments and Suggestions for AuthorsPCA is a relevant and engaging topic in teaching programming. The manuscript emphasizes the potential of PCAs, especially in improving students' programming performance. While the positive survey responses indicate that teachers are willing to adopt PCAs, the manuscript could benefit from a deeper data analysis to explore potential relationships and provide a richer understanding of the results. In addition, to strengthen the scientific contribution, it is necessary to involve more diverse and comprehensive literary sources. Expanding the literature review increases the scientific depth of the research and provides a broader context for the study.
Overall, the study is a valuable contribution to the field, and further development could include a more nuanced analysis of the survey data and an expansion of the literature review to provide a comprehensive understanding of the topic.
Author Response
First of all, we would like to thank the reviewer for their comments as we believe that have improved the paper very much. The new changes are marked in blue.
PCA is a relevant and engaging topic in teaching programming. The manuscript emphasizes the potential of PCAs, especially in improving students' programming performance. While the positive survey responses indicate that teachers are willing to adopt PCAs, the manuscript could benefit from a deeper data analysis to explore potential relationships and provide a richer understanding of the results.
Many thanks. We agree a deeper statistical study has been carried out in Section 4.3
In addition, to strengthen the scientific contribution, it is necessary to involve more diverse and comprehensive literary sources. Expanding the literature review increases the scientific depth of the research and provides a broader context for the study. Overall, the study is a valuable contribution to the field, and further development could include a more nuanced analysis of the survey data and an expansion of the literature review to provide a comprehensive understanding of the topic.
In addition to the deeper statistical analysis of Section 4.3, a new Section 2.3 has been added to include more literary sources and provide a broader context to the study. Now there are 47 references in the study, which we believe can provide a better understanding of the topic.
Round 2
Reviewer 1 Report (Previous Reviewer 3)
Comments and Suggestions for AuthorsThe authors have addressed the main points of the review; while the main point that I stressed in previous reviews (i.e. the sample of choice does not fill well with respect to the research hypothesis) still stands, the paeper has significantly improved compared to previous versions and is of sufficient quality for publication.
This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.
Round 1
Reviewer 1 Report
Comments and Suggestions for Authors=Summary
In the context of teaching Computer Programming to children,
the authors discuss in the article the use of PCAs
(Pedagogic Conversational Agents) to engage students and help
them learning programming.
In the article, the authors present and discuss an experiment
carried out with teachers (in the case, Primary Education
pre-service teachers)
that collaborate in the design of a PCA aiming at
creating friendly robots acting as children's companion
to be integrated in their classrooms to help students
learning programming.
The study reported showed that the enrollment of teachers in
the co-design of the PCA's is crucial for their acceptance as
valid/usefull teachers' helpers in the classroom.
=General Comments
The topic is interesting and important; the paper deals with a major concern of
Programming Teachers all over the world.
The main contribution of the work reported in the article is
a survey, and the lessons learned with it, before and after the
co-design of a PCA; it is a step forward the previous work of
the authors in the field that deserves further work to be implemented.
The paper is well structured and is clear, easy to read and follow
(lots of writing details need to be fixed so that the paper is
suitable for publication).
My main recommendation is that the article must provide in
section 1 or 2 more
information and details about PCA's (see comment (h) below) !
=Detailed comments
to improve the paper, please consider the following comments/suggestions:
a)lines 35-36
I don't understand this sentence
"students, 35 who need to be taught by the students, called Teachable Agents;"
I think you mean "need to be taught by the teacher", is that?
b) lines 67-68
the sentence
"we present an experience in which a co-design with pre-service Primary Education teachers to create with them a PCA to teach programming to children."
is strange! please revise it!
c) line 70
"The paper is organized into four sections: Section 2 reviews"
==>
"The paper is organized in five sections. Section 2 reviews"
d) line 84
"PCAs have been investigated since 2000 [17] with body,"
I can understand this sentence "with body" ???
e) line 132
"The programming language is simple without syntax."
This is a wrong statement!!! Scratch as all other programming languages
has a well defined and proper syntax!
please remove the sentence or change it express that being
a iconic visual language, there is no need to know by heart
or write keywords neither to know the statement structures...
f) lines 132-135
that block is strange, difficult to read due to some mistakes.
please revise it as sugestted below
The instructions are blocks that must be dragged and plugged
together like a puzzle.
Executing the program it controls the movements or other
action of a cat, as can be seen in Figure 2 that illustrates a
basic program to say Hello.
Once the “say hello” instruction is executed, the cat on the
right windows exhibits the sentence “Hello”.
g) line 146
"as executing visual programming languages on robots such as LEGO"
==>
"as using visual programming languages control robots such as LEGO"
h) lines 171-182 (Figure 4)
lots of space used in that figure to show the p-code (in Spanish),
which is useless in the present context, and actually very difficult
to understand the Dialogue (actually a very short example, in small
sized letters) that is the focus of the paper.
As a consequence, and after reading the text and the image
abou your PCA, Alcody, two main question are in my mind:
?why is it an "emotional agent" ?
?why do you call it a "learning companion"?
for me it seems to be a PCA in a Teacher Role (more than a Companion)!
I guess that my doubts derive essentially from the fact that in
the beginning a deeper explanation about PCA's and there types
and acting modes is missing!
i) line 193
"the goal here is"
==>
"the goal of the experiment describe in this article is"
i1) line 195
"Moreover, the goal is to co-design with them"
this is confusing because in the previous paragraph you have
affirmed "the goal here is" !!!
? what is indeed your goal ?
I suggest to rewrite in the following way
"To attain the referred goal, we propose to co-design with them..."
j) line 378
"4. Discussion"
One error: the section number must be 5 and not 4!!!
A suggestion: chose for this section title "Conclusion"
instead of "Discussion" -- it is very strange a paper without a
Conclusion section!
line 386
"a detailed step-by-step how-to guide"
what is it? what do you mean?
k) lines 394-395
"3. When creating an activity for the PCA to teach programming it
should include samples, tutorials, and a preview of how to solve the activity."
This is a too complex (many issues proposed) and also to vague sentence!
Please try to be more specific for each item!
Comments on the Quality of English Language
Please look at the text in the previous item
Author Response
The authors would like to thank the reviewers for their useful comments which we consider that have improved the paper very much. The writing details have been applied and more information about PCAs have been provided as indicated. Changes have been marked in yellow.
Answers to reviewer 1
to improve the paper, please consider the following comments/suggestions:
a)lines 35-36
I don't understand this sentence
"students, 35 who need to be taught by the students, called Teachable Agents;"
I think you mean "need to be taught by the teacher", is that?
In this case the role of the agent is to be a student. The agent does not have information. On the other hand, the student is the one teaching the agent.
- b) lines 67-68
the sentence
"we present an experience in which a co-design with pre-service Primary Education teachers to create with them a PCA to teach programming to children."
is strange! please revise it!
It has been rewritten.
- c) line 70
"The paper is organized into four sections: Section 2 reviews"
==>
"The paper is organized in five sections. Section 2 reviews"
Thanks, changed.
- d) line 84
"PCAs have been investigated since 2000 [17] with body,"
I can understand this sentence "with body" ???
It has been modified.
- e) line 132
"The programming language is simple without syntax."
This is a wrong statement!!! Scratch as all other programming languages
has a well defined and proper syntax!
please remove the sentence or change it express that being
a iconic visual language, there is no need to know by heart
or write keywords neither to know the statement structures...
We are sorry. It is true. The sentence has been removed.
- f) lines 132-135
that block is strange, difficult to read due to some mistakes.
please revise it as sugestted below
The instructions are blocks that must be dragged and plugged
together like a puzzle.
Executing the program it controls the movements or other
action of a cat, as can be seen in Figure 2 that illustrates a
basic program to say Hello.
Once the “say hello” instruction is executed, the cat on the
right windows exhibits the sentence “Hello”.
- g) line 146
"as executing visual programming languages on robots such as LEGO"
==>
"as using visual programming languages control robots such as LEGO"
Changed.
- h) lines 171-182 (Figure 4)
lots of space used in that figure to show the p-code (in Spanish),
which is useless in the present context, and actually very difficult
to understand the Dialogue (actually a very short example, in small
sized letters) that is the focus of the paper.
As a consequence, and after reading the text and the image
abou your PCA, Alcody, two main question are in my mind:
?why is it an "emotional agent" ?
?why do you call it a "learning companion"?
for me it seems to be a PCA in a Teacher Role (more than a Companion)!
I guess that my doubts derive essentially from the fact that in
the beginning a deeper explanation about PCA's and there types
and acting modes is missing!
Section 2.1 has been extended with a more detailed explanation of PCAs and the types and acting modes, and Section 2.2 has been improved by explaining why Alcody is an emotional learning companion.
- i) line 193
"the goal here is"
==>
"the goal of the experiment describe in this article is"
Changed.
i1) line 195
"Moreover, the goal is to co-design with them"
this is confusing because in the previous paragraph you have
affirmed "the goal here is" !!!
? what is indeed your goal ?
I suggest to rewrite in the following way
"To attain the referred goal, we propose to co-design with them..."
Many thanks, changed!
- j) line 378
"4. Discussion"
One error: the section number must be 5 and not 4!!!
A suggestion: chose for this section title "Conclusion"
instead of "Discussion" -- it is very strange a paper without a
Conclusion section!
Thanks, changed!
line 386
"a detailed step-by-step how-to guide"
what is it? what do you mean?
It has been rewritten, we just indicate that teachers need a guide to use the PCAs to teach programming.
- k) lines 394-395
"3. When creating an activity for the PCA to teach programming it
should include samples, tutorials, and a preview of how to solve the activity."
This is a too complex (many issues proposed) and also to vague sentence!
Please try to be more specific for each item!
It has been split into three different points to explain each point with more detail.
Reviewer 2 Report
Comments and Suggestions for AuthorsThe manuscript introduces a promising concept but is hindered by a lack of scientific rigour and depth of analysis. A more comprehensive presentation of methodologies, results, and critical examination of potential limitations would enhance the article's credibility and contribute significantly to the discourse on integrating PCAs in educational settings.
Author Response
The authors would like to thank the reviewers for their useful comments which we consider that have improved the paper very much. The writing details have been applied and more information about PCAs have been provided as indicated. Changes have been marked in yellow.
Answer to reviewer 2
The manuscript introduces a promising concept but is hindered by a lack of scientific rigour and depth of analysis. A more comprehensive presentation of methodologies, results, and critical examination of potential limitations would enhance the article's credibility and contribute significantly to the discourse on integrating PCAs in educational settings.
Many thanks, the other reviewers agree with you and we have applied the comments to better describe the methodology, results, limitations and conclusions of the paper.
Reviewer 3 Report
Comments and Suggestions for AuthorsThis article investigates the beliefs of a set of pre-service teachers towards the use of Pedagogic Conversational Agents (PCAs) as a tool to support students' learning of programming principles. The topic is of interest and in line with the scope of the journal. However, the study has serious limits in its conceptualization and, and the presentations is rather low-quality due to poor english usage.
The introduction does a good job of introducing PCAs and the reason for which, despite positive evidence in the literature, those tools have so far seen limited use in educational practice. However, the research question is not clearly stated. As it stands, it seems the researchers are interested in understanding whether the same problems identified in the literature for PCA also apply to the Primary school context. However, their sample is made of 44 undergraduate university students (or pre-service teachers? it is unclear). It is doubtful that such students possess the required experience to identify the needs of a primary school context to really drive co-designing of PCA features. Thus, it is not clear whether the results presented in the study can really be of use. The students were familiar with programming learning but unfamiliar with PCA. To this aim, the researchers designed a prototype to show students the potential of PCAs to drive programming learning. The results from a post- questionnaire to gather the students' reaction indicated that the students aknowledge the potential of PCAs for teaching and learning programming, but that is the only result that can be drawn from the study. THe study closes with a set of guidelines ostensibly derived from the students' feedback, which was however largely derived from multiple-choice questions where the options were provided by the researchers, plus a general "notes" open questions of which only a handful of examples are shown. Furthermore, the article completely lacks a critical discussion of the results obtained, including limitations of the study, perspectives, and relevance of the findings for advancing the field. Taken together, the methodology and conceptualization of the study are insufficient.
Comments on the Quality of English LanguageThe quality of English is poor. The text is understandable but there are too many wording or syntactic errors. For instance, take lines 79-82:
"It is very popular the simplest version of PCAs that are chatbots without artificial intelligence but able to make turns with a human user to achieve certain goal. Some samples are Alexa to listen to music or buy, the Google Assistant, Siri of Apple or the more innovative chatGPT with some AI
for Education. "
Those lines should be entirely reworked. Furthermore, there are some "false friends" like the use of "samples" instead of "examples". The authors also seem to use "university students in Education" and "pre-service teachers" as interchangeable, whereas the second denotes only the students that are already carrying out teaching activities as part of their degree course.
To put it simply, the quality of English language is at the moment insufficient for publication.
Author Response
The authors would like to thank the reviewers for their useful comments which we consider that have improved the paper very much. The writing details have been applied and more information about PCAs have been provided as indicated. Changes have been marked in yellow.
Answers to reviewer 3
This article investigates the beliefs of a set of pre-service teachers towards the use of Pedagogic Conversational Agents (PCAs) as a tool to support students' learning of programming principles. The topic is of interest and in line with the scope of the journal. However, the study has serious limits in its conceptualization and, and the presentations is rather low-quality due to poor english usage.
The paper has been rewritten to improve its conceptualization and the presentation of methodologies, results with a critical examination of potential limitations as also suggested by Reviewer 2.
The English has been reviewed by an expert in English usage to improve the quality of the paper presentation given that we are not English native speakers.
The introduction does a good job of introducing PCAs and the reason for which, despite positive evidence in the literature, those tools have so far seen limited use in educational practice. However, the research question is not clearly stated.
The research questions have now been stated both in the Introduction and in Section 3.1.
As it stands, it seems the researchers are interested in understanding whether the same problems identified in the literature for PCA also apply to the Primary school context. However, their sample is made of 44 undergraduate university students (or pre-service teachers? it is unclear). It is doubtful that such students possess the required experience to identify the needs of a primary school context to really drive co-designing of PCA features.
Thus, it is not clear whether the results presented in the study can really be of use. The students were familiar with programming learning but unfamiliar with PCA. To this aim, the researchers designed a prototype to show students the potential of PCAs to drive programming learning. The results from a post- questionnaire to gather the students' reaction indicated that the students aknowledge the potential of PCAs for teaching and learning programming, but that is the only result that can be drawn from the study. THe study closes with a set of guidelines ostensibly derived from the students' feedback, which was however largely derived from multiple-choice questions where the options were provided by the researchers, plus a general "notes" open questions of which only a handful of examples are shown. Furthermore, the article completely lacks a critical discussion of the results obtained, including limitations of the study, perspectives, and relevance of the findings for advancing the field. Taken together, the methodology and conceptualization of the study are insufficient.
We would like to thank the reviewer for the comments provided which have made us reflect on the research questions of the paper and to improve the methodology and conceptualization as also requested by Reviewer 2.
They were 44 pre-service teachers. Section 3 has been rewritten to answer all reviewer questions. In Human-Computer Interaction it is usual to show pre-designed screens to help users to choose the options they prefer as it is harder for users to choose from scratch. All in all, an open ended questions was provided to give total freedom to the participants and more answers are provided in the paper. Moreover, a Discussion is provided in Section 4 answering the research questions and with a limitations section.
Comments on the Quality of English Language
The quality of English is poor. The text is understandable but there are too many wording or syntactic errors. For instance, take lines 79-82:
"It is very popular the simplest version of PCAs that are chatbots without artificial intelligence but able to make turns with a human user to achieve certain goal. Some samples are Alexa to listen to music or buy, the Google Assistant, Siri of Apple or the more innovative chatGPT with some AI for Education. "
Those lines should be entirely reworked.
Furthermore, there are some "false friends" like the use of "samples" instead of "examples". The authors also seem to use "university students in Education" and "pre-service teachers" as interchangeable, whereas the second denotes only the students that are already carrying out teaching activities as part of their degree course.
To put it simply, the quality of English language is at the moment insufficient for publication.
We agree, given that we are not English native speakers, we have asked for help to an expert in English usage who has edited the English in all the document. In particular, the indicated sentence has been simplified and the false friends have been removed.
We have now understood the difference between “university students in Education” and “pre-service teachers”, we are sorry four our mistake as we certainly thought that it was the same, but the experience was carried out with “pre-service teachers” who have teaching practices as part of their degree course. It has been corrected in the paper.
