Lessons Learned from 10 Experiments That Tested the Efficacy and Assumptions of Hypothetical Learning Trajectories
Round 1
Reviewer 1 Report
The work discusses the 2 assumptions of LP and HLT based on 13 studies (which I assume this work is a meta-study of). While the overarching theme of this work has its strong merits, there are several issues that should be addressed before the article may be considered publishable.
Overall, the work appears to be a meta-study/review of the authors’ existing works (given citations A0 to A18 are unnamed). While this is okay, one would wonder why the project is broken into so many articles and an additional meta-study is necessary. Furthermore, it is not clear what are the differences between the studies, and this may be crucial for understanding why different conclusions are being derived. While I attribute this partially to the need for double blind review, the article should be as self-contained as possible so that the readers do not need to refer to other articles for context.
The organisation in its current form is also difficult to read. I am not too sure of the reason for having further discussions after the conclusion. There are multiple parts the logic of the text does not flow from one section to another, and it is hard to follow the argument. I strongly suggest the authors to review their argument from paragraph to paragraph to ensure the clarity in the argument.
There are multiple lines where the font sizes are different, or that the headings are labelled incorrectly (e.g., is Future research in line 541 a new section?), I would suggest the authors to review them carefully.
Author Response
Please see attachement.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
Some general and some specific comments:
P1: In the introduction you need to make it clearer that this paper is about math.
P1: What is “IES”?
P1: Define “moderate” evidence.
P2: Define “minimal” evidence.
P5 Table 1: Please present the concrete effect sizes and/or p-levels. Also, much more information is needed to properly understand what this (i.e. the 13 studies) is all about and understand the tables’ explanation.
P6: Define “mixed results”. Therefore the exact effects sizes need to be presented.
P7: Define “mixed results”. Therefore the exact effects sizes need to be presented.
P7: To understand Section 4, we first need to know the effect sizes, and second need to know more about the 13 studies.
P9 Figure 1: Some information in the second block (Level 1) is missing.
P10: Why offer so many excuses for not finding the expected results; perhaps HLT just isn’t better?
P12: Better explain the CP.
Author Response
Review #2
P1: In the introduction you need to make it clearer that this paper is about math.
P1: What is “IES”?
P1: Define “moderate” evidence.
P2: Define “minimal” evidence.
P5 Table 1: Please present the concrete effect sizes and/or p-levels. Also, much more information is needed to properly understand what this (i.e. the 13 studies) is all about and understand the tables’ explanation.
P6: Define “mixed results”. Therefore the exact effects sizes need to be presented.
P7: Define “mixed results”. Therefore the exact effects sizes need to be presented.
P7: To understand Section 4, we first need to know the effect sizes, and second need to know more about the 13 studies.
P9 Figure 1: Some information in the second block (Level 1) is missing.
P10: Why offer so many excuses for not finding the expected results; perhaps HLT just isn’t better?
P12: Better explain the CP.
Reviewer 3 Report
Thanks for inviting me to review this interesting work. I have the following major concerns thus cannot recommend an acceptance of this manuscript.
First is the IRB problem. This version has No information about ethical clearance and consent forms.
Second, the title is eyecatching but grandstanding. It is unspecific, misleading, overstating, vague. Please read the APA Publication Manual (7th) for detailed guides.
Third, the abstract is incomplete, missing important information (sample size and sampling approach, design, measures, data analysis).
Fourth, there is no literature review. It lacks definition/reviews of key concepts, IVs, and DVs; bad organized; no justification of the study.
Fifth, there are no research questions. You need to clearly define your research questions before the section on methods.
Sixth, this missing information about the methods such as the sample, sampling, measures, design, and procedure.
Last but not least, there is NO DISCUSSION.
Author Response
Although Reviewer 3's first, third fourth, fifth, sixth, and seventh points would be applicable for a research report, our submission is a review article.
In regard to the second point, we changed the title to: "Lessons Learned from 10 Experiments that Tested the Efficacy and Assumptions of HLTs."
Reviewer 4 Report
First of all, I would like to thank the authors for their contribution "Why Is It So Difficult to Confirm the Obvious? The Case of Gauging the Efficacy of HLTs".
Regarding the research base of the study, the literature review is thorough. The findings of previous research are engaged thoughtfully and analyzed critically.
In general, the discussion of results could adopt a more critical, analytical perspective to supplement and strengthen the descriptive reporting of the data.
In sum, I applaud all the efforts of the author(s) for this research and for the revised version of this manuscript.
Author Response
We appreciate Reviewer 4's positive comments. We made several changes to provide a more critical perspective of our reported data: (a) We qualified the entries for Study 8 in Table 1. (b) We added a footnote to Table 1 qualifying the external validity of our small-n studies. (c) We deleted mention of Study 8, which unlike the other 10 experiments, was a qualitative study that did not involve inferential statistics. (d) We toned down our conclusions on p. 16.
Round 2
Reviewer 1 Report
Thank you for incorporating the comments, the article is significantly easier to read for new readers in this version. Some minor comments:
L42: Is the Guide referring to the IES guide or the article?
L49: the link should be a citation.
L93: Inconsistent font.
Author Response
L42: We replaced "this Guide" with "the IES Practice Guide."
On L40, we left the web site reference in the text for the sake of simplicity. (It would have got wordy to do otherwise.)
L93: The font of "Ginsburg et al., 2008" was changed from Time-New Roman 12 to Palatino Linotype 10.
Reviewer 2 Report
The paper has improved considerably in terms of content and structure. Some points.
L 262: I think that “mixed” is better than “different”.
L 286: “different” does not feel right here. Also: “child...” ?
Table 1: The authors should be aware that both p and ES depend on n. E.g., an ES of 1.3 or even of 1.68 with an n of 15 is very questionable, in fact, ESs will be dramatically inflated in such cases (e.g., Slavin, R., & Smith, D. (2009). The relationship between sample sizes and effect sizes in systematic review in education. Educational Evaluation and Policy Analysis, 31(4), 500-506).
L 600: The authors should be aware that both p and ES depend on n. E.g., an ES of 1.3 or even of 1.68 with an n of 15 is questionable.
L 855: I do not agree at all with the qualification “strongly”; see earlier comments.
L 890: I do not agree at all with the qualification “clearly”; see earlier comments.
Author Response
L 262 & L 286: We changed "studies across different mathematical topics and different age groups" to "studies across VARIOUS mathematical topics and age groups." We were not sure what "Also: 'child...'?" meant.
Table 1 & L 600. We thank the reviewer for calling our attention to the Slavin and Smith (2009) article. We added the following footnote to Table 1: "Slavin and Smith (2009) caution that effect sizes for small-n studies, such as Study 4, 5, and 10, are more variable than those of large-n studies. Thus, the former produce less reliable and replicable estimates of program impact than the latter. They further note that the most important source of this greater variability among small-n educational studies may be to what Cronbach et al. (1980) call 'superrealization.' Superrealization refers to high implementation fidelity due to the fact that small-n studies afford better monitoring and more input by experimenters than would be available at scale. Slavin and Smith conclude that, although this variable may not impact internal validity, it can appreciably affect external validity." The only comment Slavin and Smith made regarding the relation between p and sample size was that “it takes a larger effect size to produce statistical significance in a small study than in a large study.” As this does not undermine the significance of the ps we reported for the small-n studies, we did not comment on the relation between p and sample size in the footnote or elsewhere.
L855 & L890: Slavin and Smith (2009) do not argue that a small n will dramatically inflate an ES and, thus, render it questionable. (The word 'inflates' does not appear in their article.) They do discuss a publication bias, which does not affect either internal or external validity of study and superrealiziation, which mainly affect external validity. Although we do not agree that a small n necessarily "inflates" an ES and certainly not a p, we agree with Slavin and Smith (2009) that the it renders an ES less stable and, thus, limits generalizability at scale. Therefore, as Reviewer 2 recommended, we deleted the qualifications "strongly" and "clearly."
