Redesign of a Life Cycle Figure Improves Student Conceptions of Ecology and Evolution

Round 1

Reviewer 1 Report

Overview

The manuscript reports one study examining student’s knowledge after seeing circular and linear life cycle diagram with either singular or multiple organisms per generation. The study is methodologically sound, and the data is interesting. I have wanted to do a study like this one for years and the authors did a great job! My comments below are with the hope of further strengthening the manuscript.

Major concerns

The authors should be clearer as to why the difference between linear and circular diagrams would influence student’s understanding of evolution. This could be set up in the introduction.

Likewise, the decision to include only a single organism in the adult stage (even in the diagrams with multiple organisms at earlier stages) should be explained.

One methodological issue is the lack of a Linear-Single figure in the study. This would have allowed the authors to test all of the combinations of their variables of interest. Currently, it is not possible to determine if the results are due to the diagram being linear or the interaction of being linear and showing multiple offspring. Perhaps this could be a limitation mentioned in the method section

All figures in the paper should include error bars to show the uncertainty around the data. Additionally, although I found the results to be very clear, I found the lack of inferential statistics puzzling. The authors should report whether the condition differences they show in the text and figures are statistically significant. This should be presented in the main text and not in the figure captions.

Given their sample size, I wonder if the authors could examine if the students in the class for science majors (versus non-majors) made different selections or responded differently to the diagrams. If they did or did not would be interesting! And it might help instructors to know which diagrams leads to more accurate inferences for their own students.

Minor concerns

If allowed by the IRB, the authors should consider reporting the gender and racial/ethnic breakdown of their sample.

On page 6, the note about scoring probably does not need a number

Line 162, typo “whcih”

Author Response

Thank you very much for devoting your time and expertise toward improving our manuscript. Your comments were extremely helpful (and your kind words are much appreciated). We hope that we have addressed all your concerns, described below.

• Clarify why circular and linear diagrams would influence student understanding of evolution.
  We added this explanation to section 1.5.
• Explain why we decided to include a single organism in the adult stage.
  We may have misunderstood this comment and apologize if we did. The decision was based on general traits of life cycles in commonly used textbooks (section 2.1). We reviewed this section to make sure the reasoning behind our choice is clear.
• Address the lack of a Linear-Single figure.
  That would have been really helpful! We chose the three figures to address one independent variable between each set (i.e., number of offspring between CS and CM figures, layout between CM and LM figures). We never expected the additive effect of both independent variables. We’ve addressed this in the discussion, section 4.4.
• Include error bars.
  Yes, we have added bars indicating 95% confidence intervals for all graphs that display averages. The line graphs do not present means, but percentage of student responses. We have clarified this at the start of the Results section. We have also moved statements of p-values and statistical significance to the main text as you recommended. We appreciate your comments on this oversight!
• Examine science-majors vs. non-science majors.
  We clarified (at the start of the Results section)that the groups showed no differences in responses to our survey and that these students participated in the survey during their first week in college. We did this to limit the impact of college coursework on their responses to ecology & evolution questions.
• Report sample demographics.
  We did not collect demographic information to maintain anonymity of participants. However, we have added general demographic information for the university to give a basic picture of student background.
• Fix typo.
  DONE!

Thank you again for your careful and supportive review of our manuscript. We are so appreciative for all the comments and think the paper is much stronger for it.

Reviewer 2 Report

This study focuses on unintentional messages about ecology and evolution conveyed in life cycle images.  The paper is clear, well-written, and the study design is appropriate. Mostly, I interested in having a bit more detail in some places.

I very much like the historical context the study is situated in, but would like a few references (e.g., lines 27 – 28 in the Introduction and elsewhere) to back up the historical perspective on the use of different representations.

Lines 156-157 the study by Sisk-Hilton et al is described presenting the “life-spiral.” Why wasn’t the life-spiral included in the design? Given that it was designed to demonstrate genetic variation, microevolution, and changes in phenotypes due to selective pressures, the presentation of this study at this point in the manuscript sets up the expectation that it will be examined (and win!) over the other approaches.

Lines 184 on in the Materials and Methods section – while it is pretty common to include sparse details about college age participants, I think it is helpful to have more detail about them. If the data were not collected (e.g., gender, mean age, etc.,). Where most of the participants from a few southeastern states? Or does the university draw more widely in the US and internationally? Given how certain states view the teaching of evolution, this information could be useful.

I really like how the actual study is set-up to explore a gap in the literature and the design of the stimuli.

Lines 215 and on.  More information on the pilot survey should be provided.  Who were the participants? How many were there? How were they selected? Is the difficulty level the percent of participants who answered correctly on the pre-test?  I assume so, but it is not entirely clear.

Item 2 seems somewhat problematic given that all living things die, and although death often occurs early in life, it certainly occurs in adulthood, if it has not occurred at a previous stage. Were participants somehow primed not to consider this point about all living things die?

The analyses and results are clearly presented.  Although I like the detail, information is presented in multiple formats a couple of times throughout the results section. If space is needed some of this could be cut.

Figure 3a and b are non-standard ways of showing the variability around the mean scores – but I think that it is effective.

I realize that some of the results shown in Table 3 are unexpected, but I thought there could be a better attempt to explain why they may have occurred in the discussion.

It was a bit surprising that the Sisk-Hilton et al study was not returned to in the final discussion.

Author Response

Thank you very much for devoting your time and expertise toward improving our manuscript. Your comments were extremely helpful (and your kind words are much appreciated). We hope that we have addressed all your concerns, described below.

• The first comment was the only one we were unsure of. We weren’t sure if the list of functions of life cycles (lines 27-28) required references or if the historical background needed more detail (of if these two possible interpretations were both wrong!). We thought that further historical background was more likely, so we added a paragraph in section 1.2 to address the historical shift. We wish there was more information on the history of decision making in textbook figures and considered that this could make an interesting future article.
• Concern: why the life spiral wasn’t included.
  We added information to clarify that the life spiral included multiple generations, which would add multiple variables to the treatments. We also added several references to the life spiral in the discussion.
• Concern: demographic information.
  You are correct that we did not collect demographic data from the participants. We did add a paragraph describing the demographics of the university as a whole. We also thought your comments about the regional differences in teaching evolution were extremely valuable; we added information about the treatment of evolution in the state.
• Concern: pilot study
  We expanded information about the pilot study, the number of participants, how they were selected, and more detail about the survey and how it changed from the pilot.
  And, yes, you are correct that the difficulty level is the percent of participants who answered correctly. We added that in parentheses when we first address the difficulty level.
• Concern: priming that all living things die
  All instructions and questions were the same for the three treatments, so there was no priming for any points. We tried to clarify that, given the only difference is the one variable different between the two compared figures, so that variable should be the cause of any difference in responses.
• Concern: data presentation
  We have rerun our data analyses and reorganized the data presentation to display the information in a more concise manner.
• Concern: variability in data
  We have added error bars to all graphs that display means.
• Concern: unexpected data in table
  When we reran our data analyses and reorganized the data presentation, we found an error in that table. We were relieved to have corrected the data; we also added to the discussion about the patterns seen.
• Concern: Sisk-Hilton not returned to in discussion
  We have corrected this by addressing Sisk-Hilton’s life spiral and her team’s project multiple times in the discussion!

Thank you again for your careful and supportive review of our manuscript. We are so appreciative for all the comments and think the paper is much stronger for it.

Reviewer 3 Report

The authors report on the results of a quasi-experimental study into the influence of three different formats of graphics on students' responses to a set of biological questions.  The work was well described, and the sample and methods seemed appropriate for the research questions.  While I found the central question--that is, the way graphics can shape students' interpretations--to be interesting, at points in this manuscript, I feel that the authors' interpretations outstripped their data.  I'll describe my concerns here:

1.  The authors suggest that their results speak to the ways in which different visual formats cause or support misconceptions on the part of their students.  Here the authors went beyond their findings.  As I understand it, the authors asked this set of 8 questions at one time early in the semester.  And of this small group of questions, each targeted a different question (e.g., variability, mortality rates, etc).  Measuring  students' conceptions with just one question is not sound psychometric practice.  Instead, I would assert is what is being measured here is students' interpretation of a graphic, rather than tapping into some underlying conception.

2.  I am particularly perplexed about the questions used when we consider the first (traditional graphic). The students were directed to use the graphic to answer the questions--but information that speaks to several of the questions (#1, 4, & 5) was not available in the first diagram.  Thus, there seems to be a contradiction between the information available and the directions to students.  This issue needs some sort of resolution--as the current interpretation does not suffice. 

3.  Throughout, I think the argument could be strengthened by making it more modest--that is, these data speak to students' interpretation of graphics, and not the ways in which these graphics support students' conceptions (mis and otherwise).  I think there is great utility of this argument and I think this does better justice to your data.  

Author Response

Thank you very much for devoting your time and expertise toward improving our manuscript. Your comments were extremely helpful (and your kind words are appreciated). We hope that we have addressed all your concerns, described below.

• Two of your concerns (#1 & #3) suggested limiting our conclusions to the scope of the data. We are very appreciative of these comments and agree. We addressed these comments throughout the manuscript. Three major changes included:
  - a new paragraph in the introduction defining our use of the terms “conceptions” and “misconceptions,”
  - comments on the limitations of gathering a full understanding of student conceptions from the few questions (as well as a more detailed description of our pilot study, which included more, and open-ended, questions), and
  - changes to the results section to describe the data as “responses to the questions” rather than “conceptions.”
• Concern #2 focused on the question wording and information available in the figure. We added information throughout the results and discussion about direct interpretation of the graphic compared with conceptualization of non-visible information.

Normally, we would direct you to specific locations in the manuscript that demonstrate we have addressed your thoughtful concerns. Your comments encouraged us to reconfigure much of the paper, so almost every section has been modified in ways that we hope you find improved.

Thank you again for your careful review of our manuscript. We are so appreciative for all the comments and think the paper is much stronger for it.

Reviewer 4 Report

See attached.

Comments for author File: Comments.pdf

Author Response

Thank you very much for the time and careful attention you gave our manuscript. The comments were extremely helpful (and your kind words are much appreciated). We hope that we have addressed all your concerns, described below.

• The main concerns addressed our data analysis. We are so thankful for your constructive criticism. You were absolutely correct and we apologize for the confusion we created. We have gone back over our data, reanalyzed using ANOVA in some cases and chi square tests in other cases. We have clarified when each test was used, and why some data are presented as bar graphs and others as line charts. We removed two of the tables to incorporate those data with corresponding bar graphs. In addition, with this reevaluation of our data analysis, we also found an error in one of the remaining tables and are so grateful to you for questioning the data presentation so that we could correct the error.
• You also addressed the problem with printing the graphs in black and white. We chose the background color scheme to differentiate from the red values of our data (the green was a problem for color blindness checker). We had not focused on greyscale. On your recommendation, we added horizontal bars to the green zones and black stipples to the yellow zones.
• We have also incorporated a citation and clarification for difficulty level and discrimination indices.
• We addressed all of the “minor comments” (and the 1988 BioScience article is the most recent source we could find! Many articles focus on one particular aspect of textbooks, but this is the only one that analyzed them in general. We even did a reverse search for citations and found only 3 papers referenced this one. We agree! Though, given that the textbook industry has experienced such a tremendous shift over the past decade, any comparisons would be quite the challenge.)

Thank you again for your careful and supportive review of our manuscript. We are so appreciative for all the comments and think the paper, especially the data presentation, is much stronger for it. We hope you think so too.

Round 2

Reviewer 3 Report

This is a much improved manuscript.  However, my concerns around the meaning of your findings remain:  That is, your data speak to students' interpretation of the figures NOT their conceptualization of these processes.  Your findings may have an impact on students' conceptualizations, certainly.  But your findings do not speak directly to their conceptualizations.  The work would be strengthened by staying within the boundaries of your data.  

Points where this problem emerges include:

Page 9, last paragraph) and again page 10 (last paragraph) you are suggesting that students' responses are reflective of misconceptions.  Simply put, you do not have enough data to speak to students' conceptions (mis or otherwise)….rather. your data speak to their interpretation of these graphs that confirm a particular conception.  

Again, on page 14, indicate this again here, your results speak to students’ interpretation of figures (as indicated by their responses to questions about those figures).

Page 14, your results do not speak “ reconfiguring the image to the Linear-Multiple (LM) format could produce more accurate 542 conceptions of life history and selective pressures)….You don’t have the data to speak to more accurate conceptions, but the ways in which graphics support students’ interpretation of figures.  (This change is also needed in the abstract.)

Page 18, include that there is a need of research to explore how the design of figures supports or inhibits the development of students' conception of these phenomena (as your work speaks to their interpretation NOT conceptions).

Author Response

Thank you for your continued work to help us improve our manuscript. We hope we have addressed the comments you sent.  

Comment 1: Page 9, last paragraph) and again page 10 (last paragraph) you are suggesting that students' responses are reflective of misconceptions. 
Response: We changed pg. 9 paragraph to focus on the data rather than interpreting the data as a conception (e.g., from "held a more accurate view" or "thought" to "responded with 'X'"). The last paragraph on page 10 only presents data, without reference to conceptions or misconceptions. We weren't sure where the problem paragraph was, but did review the entire manuscript again to ensure our conclusions reflected the data only.

Comment 2:  Page 14, your results do not speak “ reconfiguring the image to the Linear-Multiple (LM) format could produce more accurate conceptions of life history and selective pressures)….You don’t have the data to speak to more accurate conceptions, but the ways in which graphics support students’ interpretation of figures. 
Response: Done. We have rewritten the sentence (496-499)

Comment 3:  This change is also needed in the abstract.
Response: We rewrote the last sentence to focus on possible application of the research rather than the implication that our data affects conceptions.

Comment 4: Page 18, include that there is a need of research to explore how the design of figures supports or inhibits the development of students' conception of these phenomena (as your work speaks to their interpretation NOT conceptions).
Response: Page 18 was literature cited, but we did add a paragraph to 4.5 (future research) to address this concern.

Thank you again for devoting your time and expertise to improving our manuscript.