Learning through the Experience of Water in Elementary School Science
Round 1
Reviewer 1 Report
This manuscript is a very nice data set that describes the experience of elementary school students learning about the water cycle. Some of the strengths of this work are:
--the use of drawings and visual representations in understanding student learning in triangulation with other lines of data.
--documentation of elementary students’ conceptions of the water cycling including groundwater.
--Use of a systems-thinking lens to explore students’ understanding.
My opinion is that with some work to clarify the results, more fully connect the lessons to the data and result, and to more carefully articulate the findings, that this paper would be a valuable contribution. My comments below are meant to help strengthen and clarify the great work you’ve done.
Major points:
1) It’s unclear why you separated the analysis of the pre and post-test performance and student drawings and other assessments by classes #1 and #2 since it isn’t part of your research question. Two sections of any course often have variability, and it can be due to many factors, including the variability inherent among students in the two sections. In the Discussion you explain the difference in the class’s performance as emerging from the additional contact with the teacher in math class where she taught units of measure such as ratio, proportion and pour rate. I think it would be helpful to do one of two things: 1) make the experience of the students in the two classes (with one class getting additional instruction) part of the research questions up front, and more clearly connect the experiences the students had in math to what they are learning in the water unit (currently it isn’t clear or convincing exactly how what they learned in math was related to the water concepts that you were measuring in the drawings and other assessments), OR, 2) remove the comparison between class #1 and #2 as it is difficult to substantiate that the math class had an impact on student’s learning about water in Class #1, as the difference may be due to natural variability in prior knowledge or academic ability in the students or slight differences in the way the water content was taught or discussed in each section, and doesn’t necessarily add to your research questions. The paragraphs about transfer (lines 529-550) are highly speculative since no data were collected relating to the students’ experiences in the math course.
2) The discussion is weak in that it does not clearly connect what happened in the study to the results of the data and to your claims. This may be partly due to some disorganization or lack of clarity in the results. I give many suggestions below and point out areas of weak or unsubstantiated claims.
3) It may be useful to more fully explore in the discussion how students’ knowledge and systems thinking relate. Your data indicate that students’ knowledge increased in some areas but not all, and that many of the students were able to achieve some systems thinking. Do students need to have proficient understanding of the water cycle in order to do systems thinking?
Clarifying methods and results:
The Methods section could benefit from some reorganization. It would be easier to follow if the “data” sections were not removed from the “analysis” section. For example, it was confusing to me why the drawings the students did were lumped with “quantitative data,” which is not explained until the analysis section. The disjointedness made it difficult to follow. Also, I would recommend organizing the methods around your research questions, rather than around “qual” and “quant” which would help the reader track the purpose of each data type. There were also parts of the results that I thought would be more appropriate for the methods section, such as the introduction and description of the PCA analysis, the ANOVA and the Kappa statistic.
The section that describes Table 3 is unclear. How did you come up with these themes? From which data source? What were these themes used for in your analysis? I was left unsure about how to interpret Table 3.
Line 314-316 states there was no significant difference within the groups. Could you clarify if by “group” you mean “sections” or “pre/post-test?”
How did you calculate % increase for Figure 2? Seems like large % increases given there was no significant differences between pre and post tests.
How does Figure 2 and Figure 3 relate? These are from the same data source, correct? Figure 3 just splits the two classes? It would be useful to have them BOTH be % correct or % change. Or remove Figure 2 completely as it may not add additional information beyond what is displayed in Figure 3.
Figure 6 – what is the y-axis? Can each sub-figure be placed on the same scale? The caption does not add to understanding the figure.
Discussion/Making claims appropriate:
In the results where you describe student’s learning in a qualitative way, you often make statements such as: “students also demonstrated conceptual understanding…” (line 395), “students were able to reflect on the logic…” (line 448), “Students were also able to understand the dynamic nature…” (line 459). It would be helpful to be more exact in your statements by saying “one student,” “most students,” “some students,” or an actual percentage of students who demonstrated that skill. Otherwise it is easy to over-reach in your claims. An example in your discussion where there may be over-reach is the second sentence claiming that students developed systems thinking skills… all students?
Paragraph lines 504 to 513 needs some clarification. It’s misleading to say “most significantly” in line 504 when there was not a significant increase in their scores between pre and post. Might be better to talk about the pattern of increased performance on the test rather than try to claim that there was “significant” gains. Lines 509 (Class #1 has a greater understanding….) to line 513 is disjointed and confusing, and claims of “greater understanding” and transfer are not well substantiated.
Line 515-517 “participation in the experiential learning unit developed their understanding of the systematic structure of the water cycle” … from the description of the unit in the Methods section, it’s not clear how the experiential aspect of the learning unit connected with systems thinking in the students drawing. It appears that the students explored shoreline ecology and conducted water quality sampling, but how does that translate into systems thinking? Was there a driving question to the students explorations of the shoreline and water quality? What were the “take-aways” of the student’s experiences? Maybe more description of the learning activity would help to substantiate this connection to the data.
Line 521 – claim that “during drawing assessments learning and transfer were most pronounced.” It feels like the word “transfer” is used a bit ambiguously here, and it would be helpful to clarify to the reader more early on what is meant by transfer, and how your data revealed transfer (from what to what?).
Line 559-561 – “…evident in their construction and reconstruction of water filtration models that conceptual understanding was formed through observing multiple instances of a phenomenon…” why do you say that? What is it based on?
Lines 563-568 – NGSS SS2.C – isn’t this standard more related to weathering (which happens at the Earth’s surface only)?
Lines 613-625 It feels unclear to me how the students connected what they were learning to their local place. I’m not convinced that this was the key component of the lessons that was effective. (In fact, your discussion of Class #1 vs Class #2 suggests that maybe experiences in the math class were more important than the experiential aspect of the lessons?) It would help to have more explicit connection. Or were local and experiential elements accounted for in any of the student’s work in a way that it could be coded for? Again in Lines 660 to 668 you give more connected details about Lehrer et al’s work than you have about your own… how did the experiential lessons relate to the data collected? And how does your study relate to Lehrer et al?
Lines 627-633 You are probably not the only ones to make this case, that elementary school students can learn systems thinking. Can you couch your study in the work of others – for example the systems thinking framework of Assarf and Orion is at an elementary school level.
Limitations of the study, lines 713-720. You don’t really give limitations of YOUR study here. There are limitations of your study that you should articulate. Also, I’m not sure I agree with the claim that experiential education has been primarily been within adult ed and PD settings outside of science? Seems unlikely…. Could you explain more, and explain how this relates to your work?
Very minor items:
Second sentence of the manuscript needs more clarity and connection.
Line 117 – what do you mean by “transfer to other domains?” What domains? How?
Line 124 – extra comma after “methods”
Line 189 – “learning progressions” … do you mean a learning sequence here? Learning progressions have a specific connotation.
When quoting students and using “personal communication” it may be more helpful to be more specific – were these interviews? Classroom discussion? Assignment response? Other artifact?
Line 432- change “myself” to third person.
Line 670 – 680 – reorganize? Feels repetitive.
Author Response
Please see the attached response to reviewer comments.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
Introduction
Paragraph that begins at 52 is an abrupt shift as the paper is taking about water, groundwater specifically, and suddenly the discussion shifts to ecosystems which shifts to a larger system than originally discussed. Then the paragraph at 59 returns to water. I don't follow this shift from water to ecosystem back to water. Is the attempt to bring in all of Earth's systems? Please clarify what this paragraph is offering or else I recommend removing the paragraph as complex causal relationships are in all systems.
Another out of place sentence at line 81-82. I'm not sure of the significance of how direct experiences in the natural world decrease stress, etc connects to what has been a discussion about water? I think maybe there is a sentence missing here that ties these pieces together? It seems to come out of nowhere and doesn't align with the argument provided before or after it.
Line 115 - "...the curriculum considers the learning process of an entire unit as opposed to fragmented experiences" I'm not sure what this sentence has to do with the paper? Are other water curriculum units fragmented? Or is there some specific fragmented experience that previously occurred in the classroom where this study takes place that this sentence is referring to? Otherwise, this sentence also seems out of place. I would expect this sentence to appear in a paragraph that is arguing that water units are predominately fragmented, but that isn't the argument of this paragraph that I can tell.
Systems thinking is in a research question and appears to be a very important construct in this paper. I would expect to see more information on it in the theoretical framework for if and how elementary students understand systems and why systems thinking is important and fits into experiential learning.
Methods
I don't see the link between experiential learning and the unit and/or the data analysis? Was the unit designed with the experiential learning framework in mind? Were activities performed with this framework? Did the teacher have a workshop or PD on experiential learning? I don't see how the framework and the unit that was used are linked. How did experiential learning inform the data analysis? I would expect to see that coding and/or rubrics used for quantitative data were informed in some way by the framework?
In the methods, the authors also state they used a rubric to measure systems thinking skills? How does this tie into the theoretical framework?
Line 121 - 122 - the authors mention "visually based" and modeling. Do they mean scientific modeling or teacher's showing their students how to do something? If scientific modeling, does this tie into experiential learning? Why is it important to mention/use scientific modeling for this study? Also, I don't see the importance of "visually based instructional methods" in the theoretical framework on experiential learning.
Results
The results are presented as thought it was due to the experiential learning that helped students concept development but this isn't supported within the methods. I don't see how the authors can make this claim when the experiential learning is not embedded in the unit or data analysis. How do the authors know it was experiential learning that caused conceptual development? These links need to be better defined in the methods and results and this argument needs to be developed throughout the manuscript.
Within the graph discussion about conceptual understanding for hydrology concepts, the authors state that student learning had the greatest increase in relation to modeling but give no further information about this or about what happened during the modeling. This needs to be better explicated if it is an important finding.
After finishing the results section, I think the biggest issue is that the authors are trying to include too much data, but I don't think all of the data helps make their argument. I think the authors should focus their results on the data that most clearly supports their argument about experiential learning. As it is right now the story of how this unit did, or did not, help student learning is challenging to discern. It's true the students learned (which they tend to do regardless of whether the unit was experiential learning or based on a different learning theory) snd misconceptions may have disappeared(also we tend to see in the literature after an intervention) What's not clear in this manuscript is what specifically about experiential learning supported their learning?
Summary of comments:
Overall, the authors are trying to accomplish too much in this paper and are overloading the paper with data so that the elucidation of how this unit and learning is supporting students in developing understanding of water systems is lost. I recommend the authors re-focus their research questions, so that they either focus on experiential learning only or systems thinking within experiential learning. Focusing on both makes the paper difficult to understand and determine how well the unit supported students because there isn't enough space to pay proper attention to everything that was found. It is clear that a lot of data was collected (which is great!) but discussing everything is causing the authors to bring in ideas and concepts they haven't previously discussed (such as dynamic systems thinking) and that they used data from classroom discussions (how was this coded?). It is causing what seemed to be important points at the beginning of the paper (groundwater) that seemed a very important conceptual point at the beginning of the paper gets lost in the data analysis as the authors focus on all different kinds of water flow.
Once the paper has a clear focus, then the discussion/conclusion section should be reevaluated and written to align with the research question. Right now, it is difficult to determine if I agree with the result synthesis as the results go in too many directions.
Author Response
Please see the attached response to reviewer comments. Responses for both reviewers and editor comments have been condensed into one document.
Thank you,
Amanda Levy
Author Response File: Author Response.pdf
Round 2
Reviewer 1 Report
1) The manuscript is still disconnected between the experiential learning and other activities and the learning students did, and the results and discussion. The experiential activity seemed to be about water filtration and water quality. Yet students did not have high learning gains on the “water filtration” questions on the pre and post (the water chemistry content question wasn’t reported on). (Why do you think that was still challenging to the students?) It’s difficult to see the connection between the experiential activities and the gains that the students had qualitatively around concepts that were more big-picture. For example, the drawing assessment focused on the water cycle--how was this related to what was taught in class? In the discussion you say student drawings [of the water cycle] allows them to self-assess how their prior thinking may or may not have changed throughout the experiential water unit. Was it just the water quality and filtration exercise that were “experiential?” The manuscript discussed student difficulty in understanding groundwater flow – how was that related to the experiential activities or other activities in class? Are students connecting the filtration experiment to a more abstract understanding of groundwater flow and why or why not? I’m still unclear at how the course supported students’ understanding of the water cycle or other concepts of systems thinking. It would help quite a bit to have more clear description of the learning activities that connect to these concepts, and a richer discussion of what did and did not work.
2) Lots of formatting and grammatical errors throughout, missing or extra spaces and odd phrases that need to be deleted.
3) The methods have improved in clarity and read-ability. I think the authors could read through this and improve the flow of information some more, for example, describing the pre-test after the first mention. Also putting IRR information in the methods.
4) The results should do a better job of distinguishing between what is a trend, versus statistically significant versus qualitatively interesting to note in a few students. These distinctions should be carried out throughout the discussion in terms of avoiding over-reaching claims.
Additional comments:
Line 228-231: ADD and on the spectrum—I don’t know that this adds much to the story and seems speculative. You don’t return to this point to discuss it later, so I’d suggest removing it or incorporating the general sentiment [without disclosure about this particular set of students] elsewhere within the context of justifying experiences in nature.
Line 363 – change Table 3 to Table 4. Table 4 needs unpacking. What is there is not a clear demonstration of either how you coded, or what changed pre to post. I’m not sure how to understand or interpret what is there.
Line 352-354 “all participating students (n= 56) increased their conceptual understanding of hydrological concepts and processes” needs to be better substantiated, or the claim softened.
Line 379 “Overall, both student sections improved their test scores after participating in the unit.” By how much overall? Significantly? You say in the methods that you did a t-test, but the results of this is not mentioned for the pre and post test. T-test is a different analysis than PCA and both might be insightful. I don’t believe the PCA overall tells you that student’s performance on the post-test was significantly different than the pre-test. Right?
Also, I’m not sure that the PCA adds much beyond what you can see in Figure 2 (i.e. that hydrology, envir sci and earth sci has the greatest percent increases)? My understanding is that PCA usually used to reduce the dimensionality of a complex data set by creating new variables that do a reasonable job of containing the information from the full data set. I’m not sure why you used it here as you already have a set of concepts related to each question in the test (you don’t need new dimensions). Why not do a t-test based comparison of each test concept? And is there a reason why you would want to compare the concepts against each other? (And if so, why not a more simple ANOVA?) More explanation and justification of your approach would be helpful.
Line 434 – it would be more powerful to have pre and post drawings of the same student. You do that for Student 34, but not for Student 17 & 3.
Line 435 – mention of Class #1 having higher scores—confusing now that that comparison is no longer throughout the manuscript.
Line 422 -434 It’s a little confusing that you start with showing with the drawings that students had learning gains between pre and post, then the quantitative data show no statistically significant gains pre to post. It might be better to start with the quantitative results and say that “despite no significant gains across all of the students, we observed qualitatively that some students had improvements in XXX.”
Same point is true of the written pre and post-test. Despite no significant pre to post gains (as I recall was reported in your first submission), you highlight individual responses to open-ended question as evidence of gains. Here, also it would be more accurate to report that there was a trend of higher scores on the post-test, but it there were not statistically significant learning gains. Despite no significant gains across all of the students, we observed qualitatively that some students improved in XXX illustrated by one student’s responses on question #8…” This type of clearer writing would avoid misleading the readers.
Figure 6 is still difficult to interpret. It would make more sense if the y-axis were the same across each component, and if the two classes were combined. (Note, there are two things labeled Figure 6, I am referring to the bar charts.) It’s difficult to tell what the main point of comparison is in this figure, is it across assessment types? Or across pre and post assessments? Or would it be better to arrange this data so the reader could compare across systems thinking concepts? The figure is not mentioned in the text so it’s difficult to interpret the main point that is desired here.
Line 522 – I believe it’s more conventional to report inter-rater agreement in the methods.
Line 531-533 – Landis Koch rating is about inter-rater reliability. I’m confused at how it’s being use to demonstrate a significant increase from pre to post for class #1? Is this an error?
Line 577 – did the pre to post-test scores actually increase for every student, or is it possible that at least one student had a post-test score that went down? Just wanted to be sure I read that right. You could add more information here by reporting by how much it increased on average, and re-state that it was not significantly different (which is what I understood from the first manuscript).
Line 582 – was there a statistically significant increase in scores for both sections on the drawing assessment? That’s not how I read Table 5. I read Table 5 that if you just look at pre vs post (test type) that there is no significant difference. There is a significant difference between the two class sections. And there is a significant interaction between section and test-type.
Line 594 – I’m not sure I agree with “considerable gain” … that seems like an oversell.
Line 630 635 – I still disagree with this point. I think that this NGSS standard deals only with surface water because it relates to weathering (which happens only on the surface) and not because the writers were overlooking groundwater.
Section 5.2 Limitations of the study – I’m not sure what you wrote here is the conventional type of limitations mentioned. What’s here seems like instructional limitations rather than research limitations. I think limitations you mentioned, representability and sample size are useful to mention. You could also mention limitations of the instruments and coding frameworks you used–there could be limitations based on validity and reliability. Lines 805-812 do not address limitations and should be in a different section. Lines 814-821 sound more like something that would be in a limitations section.
Author Response
Please see attached document with all responses to reviewer comments. Thank you for your continued support and patience. Wishing you and your family stay safe and well.
Author Response File: Author Response.pdf
Reviewer 2 Report
The paper is much improved from my first review of the initial submission. There are things that still require attention:
- In lines 102 (after the Experiential model) is the sentence that begins "taken together Dewey and Kolb..." yet this is the first mention of Kolb? How has he informed the study along with your discussion of Dewey? Please revise to add this.
- Systems and systems models is an important addition/re-structing of the paper, but the paragraphs require re-ordering. Lines 127 - 130 should be a the start, then the next paragraph should be 140 - 145, and the remaining paragraphs should be re-ordered. Please have a colleague or your advisor read through this section so that the paragraphs make sense. Right now they are scattered which makes following this section challenging.
- Methods are greatly improved. Lines 360 - 363 need to be relocated to the methods. They are out of place in the results.
- The presentation of results is much improved. However, in several places the author states "some students" or "a few students" These need to be quantified. Instead of "some" state how many students this is. The same for a few so the reader knows how many students fell into these categories that you are describing.
- On line 401 it randomly ends with "pre testpre test" in blue.
- On line 470, the statement is made that "thinking was most developed in Class #1" There needs to be some thought into why this happened if both classrooms were using the same materials. Was one teacher more experienced than the other? More familiar with the content? Completed the PD? This statement needs clarification for the educational community as to why this may have occurred.
- Discussion is greatly improved. However, on line 576, there is a random "pre testOverall,"
- In the discussion, return back to Dewey and Kolb to discuss how experiential learning supports 6th grade students in learning about water. The author briefly mentions that experiential learning is predominately used with adults, but this needs to be better elucidated as to how this intervention was successful.
Author Response
Please see attached document with all responses to reviewer comments. Thank you for your continued support and patience. Wishing you and your family stay safe and well.
Author Response File: Author Response.pdf
Round 3
Reviewer 1 Report
I believe the authors adequately addressed my comments. The manuscript is improved in terms of clarity, cohesiveness, scientific process and appropriate claims.
Author Response
Thank you for your support. Please see attached response.
Author Response File: 
Author Response.docx
