Transferring Knowledge in a Knowledge-in-Use Task—Investigating the Role of Knowledge Organization

Round 1

Reviewer 1 Report

My first major concern was this statement: 'The rational is that relationships between students’ knowledge-networks, goal orientation, and transfer are independent of the absolute level at which they operate. ' This assumption implies that there would be no structural relationship differences among students with different operational levels; yet the novice-to-expert literature says that there are? Isn't the search for patterns aimed to find the link between the relationships and capabilities for knowledge transfer? If these are independent, then won't patterns be meaningless?

In the mastery goal orientation paragraph (2.2.3) it would be nice for the reader to know how many constructs are supposedly addressable by the 5 items? Is it the two items mentioned in lines 71-72 - (mastery vs performance). Is there an effect of verbalization in favor of mastery versus performance knowledge?

The formation of the network graphs requires the reader to know what is in publication [40], which 'puts off' the reader (but the explanation might be too long to include? Seems like a story of transcript coding and modeling - how is that protected from bias, etc?)

Another concern..."as measured by coherence and degree" does coherence automatically go up with more concepts mentioned? Does a kind of fluency with the language and terms determine the network graphs. And do the relationships in the final graphs represent direct or indirect imputations by the network modelers from the coded transcripts?

Line 275-276 says "For students from the unit that emphasized energy forms and transformation, the centrality of transformation in their knowledge-networks is strongly related to their success on their transfer task. But Table 4 says "No student used Transformation ideas." (sitting in the transfer unit column) This is confusing. Do you mean the centrality of transformation in their knowledge representation of the training unit (not the transfer task?) 

The network graphics seem to be about the organization of constructs and are thus about the mastery aspect. Is there a network concerning the performance aspect of this knowledge? Does the mastery-focus of the network then bias the results?

minor spelling issue: 'Desing and sample'

minor grammatical issue: 'Given the emphasize that modern...'

In the future research section, it might be interesting to consider how to elicit 'performance orientation evidence' by asking students how they would go about doing something to test their ideas about the links in their mastery knowledge network.

Author Response

Comment: My first major concern was this statement: 'The rational is that relationships between students’ knowledge-networks, goal orientation, and transfer are independent of the absolute level at which they operate. ' This assumption implies that there would be no structural relationship differences among students with different operational levels; yet the novice-to-expert literature says that there are? Isn't the search for patterns aimed to find the link between the relationships and capabilities for knowledge transfer? If these are independent, then won't patterns be meaningless?

Our Response: We acknowledge that our statement is misleading. What we meant to say is that due to students coming from two different units, there are systematic differences in how well students did on the transfer task, e.g., students from one unit scoring 50% on average and students from the other unit scoring 65% on average. However, independently of the unit, the theory suggests that better knowledge organization and higher mastery goal orientation should be related to more successful transfer. This is what we indeed find in our data. To be able to analyze the results from students from both units in one statistical model (to increase our statistical power) however, we have to standardize the measures within each group.

Changes to the manuscript:  As the sentence was apparently confusing and does not really add to the manuscript, we decided to delete it.

Comment: In the mastery goal orientation paragraph (2.2.3) it would be nice for the reader to know how many constructs are supposedly addressable by the 5 items? Is it the two items mentioned in lines 71-72 - (mastery vs performance). Is there an effect of verbalization in favor of mastery versus performance knowledge?

Our Response: The five items of the mastery goal measure all address mastery goal orientation. The transfer as sense making framework explicitly connects mastery goal orientation with transfer so we did not investigate any relationships with performance goal orientation. 

Changes to the manuscript: We added a sentence at Line 232 to clarify the issue.

Comment: The formation of the network graphs requires the reader to know what is in publication [40], which 'puts off' the reader (but the explanation might be too long to include? Seems like a story of transcript coding and modeling - how is that protected from bias, etc?). 

Our response: A full explanation of the actual procedure would indeed be lengthy and we provide a description of the rational behind the procedure beginning in line 191. 

As the construction of the actual networks and network measures purely algorithmic, bias could only come from the coding of the ideas. Here, conducted a interrater reliability analysis that showed satisfactory reliability (line 189). 

Changes to the manuscript:  We added more details to the description of the network procedure beginning in line 191.

Comment: Another concern..."as measured by coherence and degree" does coherence automatically go up with more concepts mentioned? Does a kind of fluency with the language and terms determine the network graphs. And do the relationships in the final graphs represent direct or indirect imputations by the network modelers from the coded transcripts?

Our response: Neither coherence nor degree do automatically go up with more concepts mentioned. Coherence goes up if one uses the same set of ideas consistently across phenomena in the interviews from which the networks are constructed. This reflects the idea that being able to consistently activate a set of ideas reflects what diSessa calls a concept having  larger span. Degree of an idea goes up if it is used together with many other ideas AND across contexts. In sum, this means that coherence or degree do net get up if a student just uses any more ideas. It would have to be more ideas that the students consistently uses across contexts.

The reviewer rightly points to the issue of language skills distorting the results. Therefore we looked at the correlation between students language skills and their coherence score. The resulting correlation was very small and not statistically significant and thus indicates that this concern is not a problem in our data. 

The relationships in the final graph are direct visualization of ideas degree (size of the circles representing ideas) and the strength of connection (thickness of lines) between ideas. The layout is the result of a force directed algorithm, basically a method of multidimensional scaling, that arranges ideas with similar attributes closer together.

'Changes to the manuscript: As many of the points raised by the reviewer are likely a results of a too short description of the network procedure, we extended that section in the methods part, see also our response to the previous comment, and also added respective clarifications in the discussion.

Comment: Line 275-276 says "For students from the unit that emphasized energy forms and transformation, the centrality of transformation in their knowledge-networks is strongly related to their success on their transfer task. But Table 4 says "No student used Transformation ideas." (sitting in the transfer unit column) This is confusing. Do you mean the centrality of transformation in their knowledge representation of the training unit (not the transfer task?) 

Our response:

In our study, we looked at students from two different units. One that emphasized energy transfer and one that emphasized energy forms and transformations. All students then received the same transfer task. Table 4 lists the influence  of energy ideas (energy forms, energy transformation, and transfer of energy) in students' knowledge networks from the respective units on their success on the transfer task. We found that no student that studied in the unit that emphasized energy transfer used energy transformation ideas in the interviews from which we constructed the knowledge networks. Therefore, there is no centrality of energy transformation for students that studied in the unit that emphasized energy transfer (Table 4).  Students that studied in the unit that emphasized energy forms and energy transformations however, used energy transformation quite frequently in the interviews from which we constructed the knowledge networks and we found a strong effect of energy transformation which we show in table 4. 

Changes to the manuscript: We think that the reviewer may have been confused the heading we used in table 4. Therefore we added changed the column labels in table 4 to clearly indicate that the "transfer" in "transfer unit" refers to Energy transfer and not knowledge transfer or a transfer task. Further, we revised the sentences that describe the table to make them more clear.

Comment: The network graphics seem to be about the organization of constructs and are thus about the mastery aspect. Is there a network concerning the performance aspect of this knowledge? Does the mastery-focus of the network then bias the results?

Our Response: We are honestly confused by the reviewer's comment. The network is not focused on mastery, at least not in the sense of mastery orientation. The networks describe the ideas that students used in the interviews about instances at the end of the respective energy units. 

Changes to the manuscript: We think that the very short description of the network approach in the manuscript may have caused some confusion. Therefore we extended that section to make it more clear beginning in line 191 (see also response to earlier comments about the network approach).

Comment: In the future research section, it might be interesting to consider how to elicit 'performance orientation evidence' by asking students how they would go about doing something to test their ideas about the links in their mastery knowledge network.

Our Response: Similar as with the comment before, we think that the short description may have been confusing, as we did not assess mastery aspects in the networks but with the goal orientation questionnaire.

Changes to the manuscript: See response to previous comment.

Finally, we corrected the spelling mistakes.

Reviewer 2 Report

Overall, well written and constructed, and of interest.

Some small typos:

line 14  mastery. (omit full stop)

119 Research Questions (plural)

132  Design not desing

232 In Fig 2  It's was      (incorrect use of English)

265 Fig 2 should be Fig 3 (and check in text)

271 Fig 3 should be Fig 4 and check in text.

Author Response

We corrected all the minor spelling mistakes and fixed the wrong figure labels (also in the text)