Implementing Government Elementary Math Exercises Online: Positive Effects Found in RCT under Social Turmoil in Chile

Round 1

Reviewer 1 Report

Paper is interesting and useful for teachers. It has good structure and the results are clearly presented.

Author Response

Reviewer 1.

1. Paper is interesting and useful for teachers. It has good structure and the results are clearly presented.

Authors response: Thank you for your comment. Although you have not stated any concerns about our manuscript, we have included a new conclusions section in lines 429-470. Here, we develop final ideas regarding the implementation, scope of the results, lines for future study and limitations of the current study.

Reviewer 2 Report

The subject of the article on the RCT is interesting and without a doubt, the importance and impact of the said program depend on its implementation.

Title: Make it shorter and more concise. Add context (CHILE)

Abstract:

Add a brief theoretical introduction on the importance of the subject.

Specify the context of the study (More information is necessary.) "Ministry of Education" appears but the country is missing (add Chile)

It remains to mention the study population / sample. Also, specify the level of study of the schools (primary? Secondary? University? Type of centers ...

The final synthesis of results / conclusion could be more forceful.

Introduction:

Others researches from different contexts is described in great detail in the introduction. Knowing the previous studies on the subject is important, but the theoretical framework is more than a summary of previous studies. New theoretical information must be worked on and contributed.

Restructure content. Very specific information about the study is provided on page 2 (58-71). Better to merge with page 3 (112-121).

The Journal is international and there is information about Educational Context in Chile that is not detailed.

Page 2

63-64. Why was the second year of implementation not positive? Please explain.

Page 3.

Means "BID"

Methods

Information is missing on the selection of the sample (selection criteria) and its characterization (age, course, etc.). Briefly describe the type of schools (funded school; no-funded Schools?)

The 50 teachers have received previous training about the platform? Need to explain more ...

The students were informed of the participation in the study? In what way did the students give their consent to participate? Add that information.

Discussion

There is a brief reflection on the reality of truancy. But missing the difficulty of access to the Internet that students can have. They are difficult variables to control. How were they managed in the study so that they did not interfere with the results?
There is no section of conclusions where to further develop the ideas that are reached after implementation. Also, add limitations section of the study.

Author Response

Reviewer 2.

1. Title: Make it shorter and more concise. Add context (CHILE).

Authors response: We updated the name to accommodate the context and to reduce the length. The new title reads as follows:

“Implementing government elementary math exercises online. Positive effects found in RCT under social turmoil in Chile.”

2. Abstract: Add a brief theoretical introduction on the importance of the subject.

Authors response: Thank you for pointing this out. Although we agree that this is an important consideration, we believe that by staying that this is a “unique opportunity to explore the robustness of the effects on students’ learning” we address this issue. Further, due to the length constraint for the abstract we are unable to include greater specification.

3. Abstract: Specify the context of the study (More information is necessary.) "Ministry of Education" appears but the country is missing (add Chile)

Authors response: We have added a specification of the “Chilean Ministry of Education” in line 7.

4. Abstract: It remains to mention the study population / sample. Also, specify the level of study of the schools (primary? Secondary? University? Type of centers …

Authors response: Even though we specified that participants were students in fourth grade, we have included “for elementary school” in line 8 for further clarification.

5. Abstract: The final synthesis of results / conclusion could be more forceful

Authors response: As suggested by the reviewer, the final section of the synthesis was modified to emphasize the importance of the findings. Changes can be found in lines 14-18. The revised text reads as follows:

“This effect is meaningful for four reasons. First, it has double the effect of the paper-based version. Second, it was achieved during one semester only. Third, is half that obtain with the platform for a complete year with its own set of exercises and with two sessions per week instead of one. Fourth, it was attained in a semester with a lot of absenteeism.”

6. Introduction: Others researches from different contexts is described in great detail in the introduction. Knowing the previous studies on the subject is important, but the theoretical framework is more than a summary of previous studies. New theoretical information must be worked on and contributed.

Authors response: A paragraph addressing the theoretical framework of this work was added in lines 53-57. The new text reads as follows:

“The theoretical framework for the study of ICT in schools highlights the importance of the implementation process and the context in which this implementation is situated [6]. The integration and final adoption of technological tools relies heavily on these factors. This framework has been supported by empirical evidence of the effect of practice with immediate feedback from peers and teachers and the inclusion of writing justifications for math problems [7, 8].”

7. Introduction: Restructure content. Very specific information about the study is provided on page 2 (58-71). Better to merge with page 3 (112-121).

Authors response: As suggested by the reviewer, this paragraph was moved from lines 58-71 to lines 148-162 and thus merged with further specific information regarding the study.

8. Introduction: The Journal is international and there is information about Educational Context in Chile that is not detailed.

Authors response: As suggested by the reviewer, additional information regarding the educational context in Chile was added to lines 69-73. The new text reads as follows:

“According to the UNESCO 2013 TERCE assessment, Chile has the highest national average in 6th grade mathematics in Latin America [13]. However, the Programme for International Student Assessment (PISA) test for fifteen years old students, positions Chile in 59th place out of 78 participating countries. Further, its score is not statistically significantly different from scores of countries such as Kazakhstan, Moldova, Baku (Azerbaijan), Thailand, Uruguay, and Qatar [14].”

9. Introduction - Page 2: 63-64. Why was the second year of implementation not positive? Please explain.

Authors response: An explanation for negative outcomes found during the second year of implementation was added to lines 154-155. The new text reads as follows:

“According to [19], a possible explanation for this decline is the decrease in rigor of implementation compared to the first year.”

10. Introduction – Page 3. Means "BID"

Authors response: Specification for this term was added in line 97.

11. Methods: Information is missing on the selection of the sample (selection criteria) and its characterization (age, course, etc.). Briefly describe the type of schools (funded school; no-funded Schools?)

Authors response: Information regarding selection criteria and its characterization was included to lines 177-183 and in line 191. The new text reads as follows:

“In this study, we purposively recruited 50 fourth grade classrooms from undeserved schools of several Santiago districts in July 2019; 77 percent of the participating schools are publicly funded, and 23 percent correspond to voucher schools. Selection criteria include time and space allocations for the use of ICT for math teaching; school administrators who are open to the implementation of ICT; willingness of teachers to engage in the use of ICT and to classroom visitations; and school technological infrastructure including internet connection.”

“...Average age of students ranges from 9 to 10 years old.”

12. Methods: The 50 teachers have received previous training about the platform? Need to explain more ...

Authors response: As suggested by the reviewer, information related to teachers’ previous experience and training was included in lines 191-193 and 237. The new text reads as follows:

“Participating teachers had no previous experience with the ConectaIdeas platform. Teachers in the treatment group were assigned to an initial training where they were introduced to the platform and final objectives of the project.”

“Each lab coordinator visited 12 or 13 classrooms each week and provided on the job training to participating teachers.”

13. Methods: The students were informed of the participation in the study? In what way did the students give their consent to participate? Add that information.

Authors response: Clarification regarding written consent from students’ legal guardians and administrators was added to lines 193-195. The new text reads as follows:

“The implementation team secured official written consent for participation, classroom observation and data collection from administrators and legal guardians of participating students.”

14. Discussion: There is a brief reflection on the reality of truancy. But missing the difficulty of access to the Internet that students can have. They are difficult variables to control. How were they managed in the study so that they did not interfere with the results?

Authors response: As suggested by the reviewer, a paragraph describing internet access and connectivity on participating schools was included into the discussion in lines 392-397. The new text reads as follows:

“The vast majority of Chilean urban schools have fiber optic internet connection, which allowed to convert a paper-based government program of math exercises to an online version. Further, the selection criteria applied in this project, made it possible to ensure that all participating schools had the technological infrastructure required for the use of the online platform in the classroom. During the implementation, students were able to carry out all the activities without internet connection problems.”

15. Discussion: There is no section of conclusions where to further develop the ideas that are reached after implementation. Also, add limitations section of the study.

Authors response: As suggested by the reviewer, a new conclusion sections was added in lines 429-470. Here, we develop final ideas regarding the implementation, scope of the results, lines for future study and limitations of the current study. The new text reads as follows:

“The implementation evaluated in this work has important practical implications. First, converting paper-based mathematic exercises -previously used and refined for years by the Ministry of Education- to an online platform, proved to improve the effectiveness of such exercises. This effect is doubled and its significant despite the fact that the number of sessions was reduced from twice to once per week, and that the intervention only lasted one semester. Moreover, the effect was achieved despite the social turmoil that affected the country in the middle of the semester and that increased absenteeism to levels much higher than the historical ones.

Second, in each session students were required to answer at least one open question, which included arguments of the procedures and the logic used to solve the problem. These written answers were shared with their peers, who reviewed and commented on the answers. This activity shown to have an effect on student learning. These results contribute to inform policy decisions regarding the use of existing math exercises under an online platform.

Although these findings have shown promise, there are several aspects that require further study and will be addressed in future work. For instance, studying not only the length of the written answers but how they relate to the type of question posted by the teacher. [54] addressed this issue and found that the presence of certain keywords in the question demonstrated to be relevant. However, it is necessary to further extend the study of type of questions, using topic models or the natural language processing methods. It is also necessary to analyze the type and “quality” of answers given by students and its relationship to learning.

A second aspect that needs to be further studied, is the effect of the strategy of peer collaboration through student assistants, implemented in ConectaIdeas. In each session, a platform module preselects students who are performing well to become candidates for classroom assistants. A couple of students are then selected by the teacher to be teaching assistant during the session. Students can then request help from any assistant or the teacher itself, to solve an exercise. Once the assistant is finished, students can evaluate the quality of the help received, and the teacher assistant can also evaluate how well he or she thinks the person who helped understood the explanation. Evaluating the impact of this strategy will require a different experimental study.

A third feature that is important to address is the impact of the platform on teachers’ didactic strategy. In [55, 56, 57] various classroom observation protocols are used to classify each moment of the session, and different machine learning algorithms are also used to perform automatic analysis of teaching discourse transcriptions. We have been using both methodologies to determine the impact of the use of platforms on teaching strategies. This is work in progress.

Finally, one of the main limitations of this implementation is related to its sustainability and was revealed this year during the quarantine in response to COVID-19. Although most urban undeserved schools in Chile have optic fiber internet connection, students at home have very unstable internet. In addition, a big proportion of them rely on their parents’ smartphones for internet connection. Even though the ConectaIdeas platform requires very little internet bandwidth, it does need a stable connection. Thus, the challenge is to adapt the platform to work offline and to accommodate both, the interface and the exercises, to facilitate its use on small screen devices. In a future study we will analyze an offline version of the platform for smartphones that is now being tested by students from vulnerable sectors in Chile and Peru.”

Reviewer 3 Report

The work is well articulated, but the discussion and conclusions do not adequately support the results

Author Response

Reviewer 3.

1. The work is well articulated, but the discussion and conclusions do not adequately support the results.

Authors response: Thank you for pointing this out, we have edited the discussion section by adding a new paragraph in lines 392-397. The new text reads as follows:

“The vast majority of Chilean urban schools have fiber optic internet connection, which allowed to convert a paper-based government program of math exercises to an online version. Further, the selection criteria applied in this project, made it possible to ensure that all participating schools had the technological infrastructure required for the use of the online platform in the classroom. During the implementation, students were able to carry out all the activities without internet connection problems.”

Further, we have added a new conclusions section in lines 429-470. Here, we develop final ideas regarding the implementation, scope of the results, lines for future study and limitations of the current study. The new text reads as follows:

“The implementation evaluated in this work has important practical implications. First, converting paper-based mathematic exercises -previously used and refined for years by the Ministry of Education- to an online platform, proved to improve the effectiveness of such exercises. This effect is doubled and its significant despite the fact that the number of sessions was reduced from twice to once per week, and that the intervention only lasted one semester. Moreover, the effect was achieved despite the social turmoil that affected the country in the middle of the semester and that increased absenteeism to levels much higher than the historical ones.

Second, in each session students were required to answer at least one open question, which included arguments of the procedures and the logic used to solve the problem. These written answers were shared with their peers, who reviewed and commented on the answers. This activity shown to have an effect on student learning. These results contribute to inform policy decisions regarding the use of existing math exercises under an online platform.

Although these findings have shown promise, there are several aspects that require further study and will be addressed in future work. For instance, studying not only the length of the written answers but how they relate to the type of question posted by the teacher. [54] addressed this issue and found that the presence of certain keywords in the question demonstrated to be relevant. However, it is necessary to further extend the study of type of questions, using topic models or the natural language processing methods. It is also necessary to analyze the type and “quality” of answers given by students and its relationship to learning.

A second aspect that needs to be further studied, is the effect of the strategy of peer collaboration through student assistants, implemented in ConectaIdeas. In each session, a platform module preselects students who are performing well to become candidates for classroom assistants. A couple of students are then selected by the teacher to be teaching assistant during the session. Students can then request help from any assistant or the teacher itself, to solve an exercise. Once the assistant is finished, students can evaluate the quality of the help received, and the teacher assistant can also evaluate how well he or she thinks the person who helped understood the explanation. Evaluating the impact of this strategy will require a different experimental study.

A third feature that is important to address is the impact of the platform on teachers’ didactic strategy. In [55, 56, 57] various classroom observation protocols are used to classify each moment of the session, and different machine learning algorithms are also used to perform automatic analysis of teaching discourse transcriptions. We have been using both methodologies to determine the impact of the use of platforms on teaching strategies. This is work in progress.

Finally, one of the main limitations of this implementation is related to its sustainability and was revealed this year during the quarantine in response to COVID-19. Although most urban undeserved schools in Chile have optic fiber internet connection, students at home have very unstable internet. In addition, a big proportion of them rely on their parents’ smartphones for internet connection. Even though the ConectaIdeas platform requires very little internet bandwidth, it does need a stable connection. Thus, the challenge is to adapt the platform to work offline and to accommodate both, the interface and the exercises, to facilitate its use on small screen devices. In a future study we will analyze an offline version of the platform for smartphones that is now being tested by students from vulnerable sectors in Chile and Peru.”

Reviewer 4 Report

Review of paper: RCT under social turmoil shows a positive effect of 2 online platform reusing 4th-grade math exercises of a 3 previous government program

Article submitted for the Education Sciences - Manuscript ID education-893333

Review statement

The data for this paper are misleading and there is much in the way of analysis. The authors says that they analyze “the use of an online platform in a particularly unique environment”, among other things. What is clear from the abstract is that the author is trying to do too much; and what remains unclear is where the contribution will lie.

Much of the analytical foundations section simply tells us about concepts used an online platform in a particularly environment. Instead of offering an analysis, the paper merely applies a multitude of concepts to the particular case. So we get examples that illustrate this or that concept, but not an analysis.

What did the author learn that is new about “the use of an online platform in a particularly unique environment” in this case? For that matter, the aims of the study are very difficult to understand. The “deviant case” is meant to teach us about what usually goes on! What did the author learn about phenomenon through analysing data that we don’t already know? What did the author learn that is new about method? Without an argument and an analysis that carries through the pages, the paper isn’t an article.

Author Response

Reviewer 4.

1. The data for this paper are misleading and there is much in the way of analysis. The authors says they analyze “the use of an online platform in a particularly unique environment”, among other things. What is clear from the abstract is that the author is trying to do too much; an what remains unclear is where the contribution will lie.

Authors response: While we appreciate the reviewer’s feedback, we respectfully disagree. First, the data was described in detail and the methods used for the analysis are well known in the field of education. Thus, we think that the estimated implementation effect size is not misleading in any way. Further information regarding selection criteria and sample characterization was also included to lines 177-183 and in line 191. The new text reads as follows:

“In this study, we purposively recruited 50 fourth grade classrooms from undeserved schools of several Santiago districts in July 2019; 77 percent of the participating schools are publicly funded, and 23 percent correspond to voucher schools. Selection criteria include time and space allocations for the use of ICT for math teaching; school administrators who are open to the implementation of ICT; willingness of teachers to engage in the use of ICT and to classroom visitations; and school technological infrastructure including internet connection.”

“...Average age of students ranges from 9 to 10 years old.”

Second, we think this study makes a valuable contribution to the field because demonstrates how implementing paper-based mathematic exercises into an online platform improves the effectiveness of such exercises. It also corroborates how the use of written open-ended questions and the inclusion of immediate feedback from teachers can positively impact students’ gains. Particularly, the study focused on underserved Chilean schools that have been systematically neglected. Moreover, these findings were found under an unforeseen environment, which caused enormous and unexpected absenteeism during the last period of the implementation. This demonstrates the robustness of the treatment -as implemented in this study.

2. Much of the analytical foundations section simply tell us about concepts used an online platform in a particularly environment. Instead of offering an analysis, the paper merely applies a multitude of concepts to the particular case. So we get examples that illustrate this or that concept, but not an analysis.

Authors response: A new conclusions section in lines 429-470 was added to the manuscript. Here, we develop and analyze final ideas regarding the implementation, scope of the results, lines for future study and limitations of the current study. The new text reads as follows:

“The implementation evaluated in this work has important practical implications. First, converting paper-based mathematic exercises -previously used and refined for years by the Ministry of Education- to an online platform, proved to improve the effectiveness of such exercises. This effect is doubled and its significant despite the fact that the number of sessions was reduced from twice to once per week, and that the intervention only lasted one semester. Moreover, the effect was achieved despite the social turmoil that affected the country in the middle of the semester and that increased absenteeism to levels much higher than the historical ones.

Second, in each session students were required to answer at least one open question, which included arguments of the procedures and the logic used to solve the problem. These written answers were shared with their peers, who reviewed and commented on the answers. This activity shown to have an effect on student learning. These results contribute to inform policy decisions regarding the use of existing math exercises under an online platform.

Although these findings have shown promise, there are several aspects that require further study and will be addressed in future work. For instance, studying not only the length of the written answers but how they relate to the type of question posted by the teacher. [54] addressed this issue and found that the presence of certain keywords in the question demonstrated to be relevant. However, it is necessary to further extend the study of type of questions, using topic models or the natural language processing methods. It is also necessary to analyze the type and “quality” of answers given by students and its relationship to learning.

A second aspect that needs to be further studied, is the effect of the strategy of peer collaboration through student assistants, implemented in ConectaIdeas. In each session, a platform module preselects students who are performing well to become candidates for classroom assistants. A couple of students are then selected by the teacher to be teaching assistant during the session. Students can then request help from any assistant or the teacher itself, to solve an exercise. Once the assistant is finished, students can evaluate the quality of the help received, and the teacher assistant can also evaluate how well he or she thinks the person who helped understood the explanation. Evaluating the impact of this strategy will require a different experimental study.

A third feature that is important to address is the impact of the platform on teachers’ didactic strategy. In [55, 56, 57] various classroom observation protocols are used to classify each moment of the session, and different machine learning algorithms are also used to perform automatic analysis of teaching discourse transcriptions. We have been using both methodologies to determine the impact of the use of platforms on teaching strategies. This is work in progress.

Finally, one of the main limitations of this implementation is related to its sustainability and was revealed this year during the quarantine in response to COVID-19. Although most urban undeserved schools in Chile have optic fiber internet connection, students at home have very unstable internet. In addition, a big proportion of them rely on their parents’ smartphones for internet connection. Even though the ConectaIdeas platform requires very little internet bandwidth, it does need a stable connection. Thus, the challenge is to adapt the platform to work offline and to accommodate both, the interface and the exercises, to facilitate its use on small screen devices. In a future study we will analyze an offline version of the platform for smartphones that is now being tested by students from vulnerable sectors in Chile and Peru.”

3. What did the author learn that is new about “the use of an online platform in a particularly unique environment” in this case? For that matter, the aims of the study are very difficult to understand. The “deviant case” is meant to teach us about what usually goes on! What did the author learn about phenomenon through analyzing data that we don’t already know? What did the author learn that is new about method? Without an argument and an analysis that carries through the pages, the paper isn’t an article.

Authors response: The introduction section was modified so that the context of Chilean education was included and to clarify the argument and theoretical framework for this work, see lines 53-57 and 69-73. Further, the new added conclusion section addresses the aims, contributions and limitations of this study, see lines 429-470.

As stated in the manuscript and particularly in the conclusions section, the main learning of this study relates to robustness of the implementation impact under an unforeseen environment, which caused greater general absenteeism. Therefore, differences between treatment and control should have been greatly attenuated. However, even under this enormous and unexpected absenteeism in the last period of the implementation, the positive effect is still sustained. This means that the treatment is robust enough, and that implementing paper-based math exercises shows promising results for elementary students from vulnerable contexts. We believe that these results contribute to inform policy decisions regarding the use of existing math exercises under an online platform.

Round 2

Reviewer 2 Report

The article has been greatly improved and they have followed the indications of the review. For my part, it can be published.

Author Response

Reviewer 2:

Thank you for your comments and feedback.

Reviewer 3 Report

the conclusions are adequately supported

Author Response

Reviewer 3:

Thank you for your comments and feedback.

Reviewer 4 Report

Review of paper: RCT under social turmoil shows a positive effect of 2 online platform reusing 4th-grade math exercises of a 3 previous government program

Article submitted for the Education Sciences - Manuscript ID education-893333

Review statement

Same version as before!

The data for this paper are misleading and there is much in the way of analysis. The authors says that they analyze “the use of an online platform in a particularly unique environment”, among other things. What is clear from the abstract is that the author is trying to do too much; and what remains unclear is where the contribution will lie.

Much of the analytical foundations section simply tells us about concepts used an online platform in a particularly environment. Instead of offering an analysis, the paper merely applies a multitude of concepts to the particular case. So we get examples that illustrate this or that concept, but not an analysis.

What did the author learn that is new about “the use of an online platform in a particularly unique environment” in this case? For that matter, the aims of the study are very difficult to understand. The “deviant case” is meant to teach us about what usually goes on! What did the author learn about phenomenon through analysing data that we don’t already know? What did the author learn that is new about method? Without an argument and an analysis that carries through the pages, the paper isn’t an article.

Author Response

Reviewer 4:
1. Same version as before!

Authors response: While we appreciate the reviewer’s feedback, we respectfully disagree. As noted by other reviewers of this study, the article was improved based on the received feedback. Specifically, a new conclusion section was added to the manuscript. Here we develop and analyze practical implications of the study, lines for future study and limitations.

2. The data for this paper are misleading and there is much in the way of analysis. What is clear from the abstract is that the author is trying to do too much; an what remains unclear is where the contribution will lie.

Authors response: as authors of this study we don’t see why the data would be misleading. A third-party agency, MideUC, was appointed to design, administrate and evaluate pre- and post-test assessments. MideUC has great prestige in Latin America and has also participated in previous projects developed by UNESCO. Moreover, the study follows implementation criteria proposed by [3] and the US Department of Education recommendations [33] for missing data.

3. The authors says they analyze “the use of an online platform in a particularly unique environment”, among other things.

Authors response: Throughout the article we argue that the social turmoil generated a unique opportunity to test the robustness of the effects. Particularly, we indicate that: “A large unexpected disruption with huge absenteeism occurred in the second half of the semester, which turned this study into a unique opportunity to explore the robustness of the platform’s effects on students’ learning.”

4. What is clear from the abstract is that the author is trying to do too much; and what remains unclear is where the contribution will lie. Much of the analytical foundations section simply tells us about concepts used an online platform in a particularly environment.

Authors response: During the revision process, the abstract was modified in order to detail the contributions of this article: “This effect is meaningful for four reasons. First, it has double the effect of the paper-based version. Second, it was achieved during one semester only. Third, is half that obtained with the platform for a complete year with its own set of exercises and with two sessions per week instead of one. Fourth, it was attained in a semester with a lot of absenteeism”.
Moreover, we explicitly explain what, in our opinion, is the main contribution of this work: “The main contribution of this paper is to measure the effect of a platform when a completely new set of exercises are exclusively used, or from another point of view, when a set of materials with exercises is used in an online platform. Moreover, the social turmoil during the second half of the implementation period had a huge impact on attendance, which turned this study into a rare opportunity to estimate the robustness of the effect of the intervention under difficult contextual circumstances.”

5. Instead of offering an analysis, the paper merely applies a multitude of concepts to the particular case. So we get examples that illustrate this or that concept, but not an analysis.

Authors response: It is not clear to the authors what the reviewer considers an analysis to be. The article provides methods, results and discussion sections that detail the developed analysis. In order for the authors to directly respond to the reviewer’s concerns, it would be necessary a more explicit explanation from the reviewer. In this new review/comments we do not see any clarification that can help us resolve the reviewer’s concerns.

6. What did the author learn that is new about “the use of an online platform in a particularly unique environment” in this case? For that matter, the aims of the study are very difficult to understand. The “deviant case” is meant to teach us about what usually goes on! What did the author learn about phenomenon through analysing data that we don’t already know? What did the author learn that is new about method? Without an argument and an analysis that carries through the pages, the paper isn’t an article.

Authors response: We have performed an RCT that has several innovations over previous RCT studies. We have also explained that the environment in which it was carried out allows us to conclude that the effect is robust. This is new and unique. We have also detailed the effect of students writing when they justify their answers to math problems.
We do not agree that the paper is “not an article”. We have described in detail the multiple contributions of this work, and the deep and rigorous analysis carried out. We understand that other tests could be performed. Therefore, we added limitations to our study where we acknowledge the need for future analyzes, which we intend on doing. Specifically, a more in-depth analysis of the answers to open questions and on the effect of peer review.