Technology Implementation in Pre-Service Science Teacher Education Based on the Transformative View of TPACK: Effects on Pre-Service Teachers’ TPACK, Behavioral Orientations and Actions in Practice

Round 1

Reviewer 1 Report

Thank you for this interesting article relating several technology-integration related variables, such as TPACK on action, TPACK in action and affective variables. The novelty of this article, from my point of view, lies in the combination of the SAMR and ICAP model as a basis for a TPACK rubric which I think of as an interesting take. I think the article has merit and adds to the existing body of literature. However, before publication, I think there are a few points that will make the article more concise and clearer in its communication. I have made section-by-section comments below to make my points. I am looking forward to reading a revised version of the manuscript.

Section 2.21: It is not clear to me why self-reported TPACK should be combined with more objective type of measures. Please provide some arguments for why such measurements should be combined instead of just using an objective measurement.

Furthermore, there are some recent measures for TPACK on action or TPACK on action-related variables, such as 

von Kotzebue, L. (2022). Beliefs, Self-reported or Performance-Assessed TPACK: What Can Predict the Quality of Technology-Enhanced Biology Lesson Plans?. Journal of Science Education and Technology, 31(5), 570-582.

or

Große-Heilmann, R., Riese, J., Burde, J. P., Schubatzky, T., & Weiler, D. (2022). Fostering Pre-Service Physics Teachers’ Pedagogical Content Knowledge Regarding Digital Media. Education Sciences, 12(7), 440.

 

Line 50: none of these studies goes beyond => go beyond and address

 

Section 3.1: Please provide more information about the context of your study. For example: What is meant by Bachelors program, is it the same for primary science and secondary teachers? Where is this programe situated? How many lectures are there?

Section 4: Research question 2: I am not sure what the authors mean by "how pronounced". Furthermore, I think the sentence "Only reliable and valid instruments were used throughout the study" is a very bold statement considering the ongoing discussion about the validity of self-report measures, especially when it comes to TPACK. If you think the self-reported measures are all valid (what types of validity?), please provide some arguments here or link to the relevant sections in the results section.

Section 5.1.: Please provide some information why the dropout rate of the study was almost 50%. I think it is OK and I appreciate the honesty, but it should be explained here.

Section 5.2.: Sad to here that it is not possible to link the different data sources. Is it possible to link the group scores with the individual scores?

Section 5.2.1: Please also provide Cronbachs Alpha for the TPB-items.

Section 6.1.1: In table 6, please provide some information why you report Cohens d and r as effect sizes for the t-test. It is not sensible to me why the correlation coefficient should be reported here. 

Section 6.1.2.: Please provide some information about the scales. For example, what does a high value of subjective norms mean?

Section 6.1.3. Please provide information about whether the correlations were significant and provide the correlation coefficients.

Section 6.2.2. Does it really make sense here to calculate the mean values of the SAMR and ICAP - do the authors think that the levels of SAMR and ICAP are indeed interval scaled so a calculation of a mean makes sense? If yes, please provide some information why you think this calculation makes sense. If no, please consider other types of statistics (e.g. median?).

Section 7: The authors write: "Participation in the modules led to a significant 549 increase of pre-service teachers’ professional knowledge in all TPACK on action dimen- 550 sions with moderate to large effect sizes". I think it is a problem here that self-reported measures are directly labeled as knowledge, given the validity problems discussed above. Please add self-reported knowledge or such.

The authors write: "Several authors outline that the ToPB is a valid 569 framework to study pre-service teachers’ behavioral orientations towards the use of tech- 570 nology in practice [20–22,26], and can therefore be seen as an indicator for their TPACK in 571 action." Please provide some further information why you think behavioural orientations are an indicator for their TPACK. For example, I really would like to learn sailing - however, my knowledge about sailing is really limited. In fact, maybe my behavourial intention to sail might even decrease if i would know all the difficulties to do so. Please elaborate in this section what led you to this conclusion.

Section 8: Please elaborate about the jingle-jangle issue of TPACK here also. I am more than fine with using self-reported measures in studies and TPACK-studies have a quiet long tradition to do so, but it is problematic if one treats self-reported TPACK as the actual knowledge of teachers.

There are some minor spelling errors and some minor translation issues i guess, such as "pronounced".

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

The study presents novel findings that contribute to the existing knowledge in the field. The applied mixed-method research design adds significant value. Overall, I think it is a well-written paper.

However, I have some comments and suggestions to further improve the paper's quality.

In the abstract, there is no mention of the results obtained from the analysed lesson plans. I recommend adding a sentence or a few sentences regarding these results.

Regarding the use of the term "highly significant," it is unnecessary to include the word "highly." This comment pertains to both the abstract and its appearance throughout the paper. Instead, I suggest focusing on reporting and addressing effect sizes. Effect sizes are better indicators than significance levels for expressing the degree of changes, as significance levels depend on sample size.

There is a statement in lines 122-124: “In order to get a representative assessment of (pre-service) science teachers' TPACK …” The term "representative assessment" might not be suitable in this context, as it is more closely associated with sampling procedures. In a similar context, the authors use the phrase "gain an in-depth understanding" (lines 182-184), which expresses the intended meaning more appropriately.

Line 281: "Only reliable and valid instruments were used throughout the study." I recommend deleting this sentence as the aspect of using reliable and valid instruments should be demonstrated through the study's methodology (as the authors did) rather than stated explicitly. Additionally, this sentence disrupts the paragraph structure as it represents a single paragraph. 

Providing sample items would support more evidence for validity. Although the authors mentioned that “A comprehensive overview of all items can be found in [8].” (line 325), it would be beneficial to present sample items within the paper itself. This inclusion would aid readers in understanding the exact nature of the measurements, such as inserting a table with one sample item for each scale.

For the CFA analyses, it appears that the indicators are close to the acceptance level. You might revisit this analysis and explore ways to improve the model fit, such as examining modification indices. 

Alternatively, you could try different models that make sense such as the integrative view of the construct: “TPACK arises from the integration of all TPACK dimensions (CK, PK, TK, PCK, TCK & TPK).” Based on the reported reliabilities, it seems that the authors opted for the transformative view. Anyway, it might be worth even adding this aspect as an additional research question (I leave the decision to the authors). 

The authors wrote that digital reminders were provided. Although I have a picture in my mind of what could have this been, it would be useful to provide more information about this. 

In the statement that participants could "deepen their theoretical knowledge in practice by designing and teaching a technology-enhanced lesson in groups of three to four students" (lines 250-251), it is unclear whether they taught each other throughout the course or if they had to visit schools and engage in teaching practice among students in a "real" situation with some supervision.

For Tables 6 and 7, I suggest adding an explanation of the abbreviations in the title for greater precision. Both tables include relevant statistical indicators, including effect sizes for the differences between the first and second measurement points, as well as the second and third measurement points. In the text, these indicators are reported concerning the first and third measurement points. However, only the range of the effect sizes is mentioned. I recommend reporting all the effect sizes. To address this, using all three rows in Tables 6 and 7 could be a suitable solution. The first row would correspond to the difference between the first and second measurement points, the second row would correspond to the difference between the second and third measurement points, and the third row would correspond to the difference between the first and third measurement points.

In the section titled "6.1.3. Correlations between TPACK on Action and ToPB Components," I suggest including a correlation matrix for a more informative presentation.

Line 449: “…ranging from two to 14 per lesson plan.” it might be better this way: “…ranging from 2 to 14 per lesson plan.” 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

On p12 - and elsewhere - you have the phrase 'highly significant' in the results. Please be careful here - yours results are significant as shown by the statistical test or not. There are no degrees of significance. However, you may mean a widespread outcome demonstrating significance in a wide number of areas or significance across the board at 0.001 level, for example. This is a minor detail and the only thing I'd change.

 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

I thank the authors for adressing my requests and comments, I appreciate the effort the authors put in to improve the article in this short amount of time.

I appreciate taking up the idea of  Ertmer and Ottenbreit Leftwich, which now makes it sensible why both perspectives should be taken up.

I understand that it was not possible for you to link both data sets, I agree it would've been great to deepen the understanding. I also see the challenges however! I guess it would've been possible to introduce an "outside person" in the study design that acts as a middle man - students could mark their lesson plans with their code (to link it with the questionnaire) and hand it to a third person from outside. This person than has a list with codes and names and after the study does the linking between lesson plans and questionnairs, so the researchers cannot make a connection between lesson plans and questionnaires.

Indeed, r is the appropriate effect size for wilcoxon tests.

Thank you very much for your elaboration on why you chose the mean instead of the median here and i understand your choice. I think it would help readers if you would mention these arguments in 1 or 2 sentences in the article, since maybe readers may also wonder why you chose this approach. This is the only point which needs to be adressed before publishing now from my point of view.

regarding response 14: Thank you very much, i was indeed referring to section 9, sorry for that. I appriciate the authors effort to distinguish between these two types of TPACK, which is one of the strengths of this article.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf