Review Reports - Teacher Digital Literacy: The Indisputable Challenge after COVID-19

Round 1

Reviewer 1 Report

Author Response

new version of the article is attached

Reviewer 2 Report

This article is relevant and of interest, though unfortunately the level of the English and grammar make large sections of it inaccessible, and very difficult to determine you’re meaning in many places. Please consider getting peer-review from a native English speaker before any resubmission, detailed comments provided below on introduction, materials and methods and results.

Abstract

Paradigmatic methodology – what does this mean?

Line 18 digital skill should read skills (through-out manuscript)

Line 19 don’t follow the abbreviation of ACDC from Analysis of Common Digital Skills?

Line 22/23 doesn’t make sense

Introduction

Overall not really defined what digital literacy or skills are – or in particular which ones were very rapidly required during the pandemic – absence of any real discussion of the how these might have changed over and above usual digital literacy that is required of teachers.

Line 40: use a set of numerous tools to give class online. Incorrect English

Line 42 missing “a” before virtual change “was” to “were”

Line 44: remove capitals fro Emergency Remote Teaching

Line 46: I’m not sure revolution is the correct term for what happened!

Line 48: expand what were these ICT interventions – needs expanding

Line 59: communicative is not the right word here

Line 60: evidence – should be evident – lines 59-61 does not make sense

Line 63 – typo should be objective

Line 83 – fragment – expand what do you mean here?

Check figure 1 for plagiarism I’m not sure this can be used without appropriate permissions from the author

Lines 129-131 – this figure (if it can be used) needs much more introduction and description

Line 135 what is the INTEF framework? – You need to describe this in detail particularly as you are going to refer to it in your results.

Line 158 does not make sense

Lines 163-165 – verbose not clear what you mean here

Materials and methods

Not quite sure of the relevance of the study design to assess digital literacy when it was focussed between 2016-2019 – surely if you’re assessing the digital skills required during the pandemic the data needed to also include this period at the very least?

Design of study needs more detail – eg the ACDC questionnaire needs describing in more detail

197 – poor referencing style (and this kind of error is repeated through-out)

Sample size – was this sufficiently large enough if you have undertaken statistical analysis from the data?

Lines 212-213 do not make sense

Lines 227-229 do not make sense

Unclear what Figure 2 is describing?

Lines 235-243 – what is the correlation study you are referring to

246 – did you introduce bias in your study by choosing respondents that were easily accessible – did this mean that would have easy access to internet /survey monkey – did you consider posting the survey out to reduce bias?

Line 252 can you describe the debugging of data its sounds like you might also have lost some valuable response though – people that were not used to completing surveys might have been some of those with some of the lowest digital literacy that you actually needed to collect data from

Results

Line 280 – reference figure

Author Response

Comments to the section “Introduction”

(1) You begin your section 1.2 with a sentence that refers to a graphic about "the 21 digital skills". In addition, you do not have a textual elaboration of your graphic, the graphic itself is the commentary. This is a rather unusual way to start a new section. I would prefer a paragraph that explains the graphic and refers to it. It's much easier for a reader to gather the information you want to share with them.

We have added an explanation of The Common Framework for Teaching Digital Competence, including about the five areas that integrate this vision of the digital competence. On the other hand, it has been clarified what is INTEF, National Institute of Educational Technologies and Teacher Training, is an institution belonging to the Ministry of Education, Culture and Sport (MECD) of the Spanish government.

Comments to the section “Materials and Methods”

(3) Instruments section: The ACDC instrument should be described in more detail. What is the instruction text of the ACDC? Due to the exclusive availability of the ACDC in Spanish, it would be helpful if you could list all items in an appendix or supplementary file. What is the structure of the ACDC (what is the factorial structure of the 94-items ACDC?)? What exactly does it assess? How does the ACDC relate to the 5 competencies (21 digital skills)? Is there an assignment of the competences measured by the ACDC to the skills (operationalization)? Finally, please provide all Cronbach’s Alphas (or other reliability indices) for all factors of the ACDC. I hope the ACDC was developed state-of-the-art. For me, it makes no sense to calculate a Cronbach’s alpha over all 94 items (or does it mean that all items measure the same construct?). Unfortunately, it seems that the ACDC is only a collection of one-item measurements of competencies without deeper analysis of the measurement instrument.

(4) I hesitated after reading your information about grouping ACDC items by the 5 digital skills. This would strengthen my idea of the construction of the ACDC as a list of items that are not related in a diagnostic sense (no multiple item rating of different skills). Then the mean value of the grouped items was calculated. Do the items measure the same construct? How was this demonstrated? Is there a Cronbach's Alpha calculated per grouping?

We have made an in-depth description of the ACDC questionnaire, detailing the steps of its design and final validation. Images have been uploaded with the questions asked, their answer options and the presentation of the same to the participants. The Cronbach's Alpha coefficients are presented for the different groups of variables by area, by use and use.

(5) I was not happy about the use of the term correlations when you described that you used chi-square analysis. It seems that you only test for distributions of data that indicate dependencies between variables, but not how they are correlated. So I thought you would calculate correlations and use item scores or the calculated means. However, it is not clear how you calculated the chi-square analyses. What categories did you use for your analyses. I am quite confused about this. Please describe your analyses in more detail.

We believe that with all the new information presented in the “Results” section, the correlational analysis will become clearer.

(6) Could you provide further information about the representativeness of your sample? It seems that you have information about teachers working in Spain. So how does your sample differ from this population?

We have added new information in order to clarify Population and Sample

Comments to the section “Results”

(7) You have calculated correlations between knowledge and usage items. So I think you have calculated 47 correlations, but you only present a correlation of 0.9666. So how did you calculate this correlation? Anyway, why don't the ACDC characteristics determine whether you link items of knowledge and items of usage? I make the hypothesis that the ACDC is not a measurement in the psychometric sense, but a simple collection of items for descriptive purposes. Furthermore, what is shown in Figure 2, if you have combined the items of knowledge and use?

(8) I do not understand your analysis plan. Why do the things you do? What variables do you select for the analyses (items, averages, frequencies, reduction of information to the level of competence, etc.)? Why do you present some analyses at the reduced level of the overall skills (A1 to C2), why do you present some analyses at item level and why do you present some results at the level of the 5 skills?

(9) You calculate many tests. Did you have the alpha accumulation factor in mind? It seems that you are just picking the pearls of all possible calculations. Also, it is not transparent what you calculated in the section "Results of the correlation analysis". None of this is transparent. We do not get any information about the number of categories you use in chi-square analysis. We also get no information about the distribution of scores, thus we readers cannot follow your descriptions of score.

(10) I would recommend to report on effect sizes. With this large sample size you can easily get significant results, but are these results meaninguful? I would also recommend that you report correlations between gender, age, etc. Perhaps a multiple linear correlation analysis would have been more useful to identify relevant factors related to competences. But this would not be useful if you really have the shear item scores for analysis and not the multiple item competence measures.

The way to group the variables has been clarified, specifying the values of the centralization statistical variables, including the arithmetic means, from which the Pearson's Correlation Coefficient is obtained. Figure 2 (now Figure 4) has been explained in more detail.

It is clarified why we have the levels "Very low level" = 1, "Low level" = 2, "High level" = 3, or "Very high level" = 4, and levels A1 to C2. The presentation of results is adapted to the levels proposed by the INTEF, National Institute of Educational Technologies and Teacher Training.

Several examples of Chi-Square Test calculations are shown. In fact, we think that the article is excessively loaded with so many tables, but we understand the need to clarify where the results come from.

Comments to the section “Discussion and Conclusion”

(11) The part "Training proposal for the training of teachers based on their digital literacy level" should be introduced into the discussion. It is a non-specific and abstract proposal that does not contain specific training methods. It also deals with a general approach based on the competence deficits identified. This part should be more concise.

(12) The discussion does not deal with restrictions. This is a very serious disadvantage. Given the unclear methodology, the discussion is very fragile.

We have added a paragraph with some limitations on access to the sample, and added our future work plan.

Reviewer 3 Report

Dear authors,

it was very interesting to read your work. You start with a concise and transparent theoretical part in which you show why your work is about sustainability and why the topic is an international one. But I am reluctant to admit that I have great doubts about your methodological part and your results part. Finally, there is a section on limitations missing from the discussion. Here are my comments.

Comments to the section “Indroduction”

(2) A period is missing in line 175 after “submitted”.

Now I will go into the serious flaws of your paper from my point of view.

Comments to the section “Materials and Methods”

(4) I hesitated after reading your information about grouping ACDC items by the 5 digital skills. This would strengthen my idea of the construction of the ACDC as a list of items that are not related in a diagnostic sense (no multiple item rating of different skills). Then the mean value of the grouped items was calculated. Do the items measure the same construct? How was this demonstrated? Is there a Cronbach's Alpha calculated per grouping?

Comments to the section “Results”

Comments to the section “Discussion and Conclusion”

(12) The discussion does not deal with restrictions. This is a very serious disadvantage. Given the unclear methodology, the discussion is very fragile.

Other comments

You should use more international papers in English language besides your Spanish literature. This would make it easier for your international audience to follow your presentation and to immerse themselves more deeply in it.

Best regards

Reviewer

Author Response

Abstract

Paradigmatic methodology – what does this mean?

Sentence has been modified

Line 18 digital skill should read skills (through-out manuscript)

The whole text has been modified

Line 19 don’t follow the abbreviation of ACDC from Analysis of Common Digital Skills?

We have changed “Skills“ by “Competence”.

Line 22/23 doesn’t make sense.

The sentence has been re-written.

Introduction

Line 40: use a set of numerous tools to give class online. Incorrect English

Line 42 missing “a” before virtual change “was” to “were”

Line 44: remove capitals fro Emergency Remote Teaching

Line 46: I’m not sure revolution is the correct term for what happened!

Line 48: expand what were these ICT interventions – needs expanding

Line 59: communicative is not the right word here

Line 60: evidence – should be evident – lines 59-61 does not make sense

Line 63 – typo should be objective

Line 83 – fragment – expand what do you mean here?

All suggestions have been solved out.

Check figure 1 for plagiarism I’m not sure this can be used without appropriate permissions from the author

The content of this figure has been taken from The Common Framework for Teaching Digital Competence, included in the original document of INTEF, National Institute of Educational Technologies and Teacher Training. We have designed it in a more understandable way.

Lines 129-131 – this figure (if it can be used) needs much more introduction and description

Line 135 what is the INTEF framework? – You need to describe this in detail particularly as you are going to refer to it in your results.

We have added an explanation of The Common Framework for Teaching Digital Competence, including the five general areas that comprise the digital competence. On the other hand, it has been clarified, what is the INTEF, National Institute of Educational Technologies and Teacher Training, is an institution belonging to the Ministry of Education, Culture and Sport (MECD) of the Spanish government.

Line 158 does not make sense

Lines 163-165 – verbose not clear what you mean here

Sentences have been re-written.

Materials and methods

We actually have data collected to date, but by deliberate choice, only the data up to the time of the start of the confinement were taken. During confinement, the teaching staff has necessarily improved certain aspects, which are currently under study. But it was necessary to have the real image of the levels of digital competence, without this totally unexpectable situation, which may show some improvement in some areas of the digital competence.

Design of study needs more detail – eg the ACDC questionnaire needs describing in more detail

Sample size – was this sufficiently large enough if you have undertaken statistical analysis from the data?

Lines 212-213 do not make sense.

We have change the text to clarify it.

Lines 227-229 do not make sense

We have made some changes and we have relocated the text, in “Results”, where It will be better understood.

Unclear what Figure 2 is describing?

Figure 2 (now Figure 4) has been explained in more detail.

Lines 235-243 – what is the correlation study you are referring to

Several examples of Chi-Square Test calculations are shown.

246 – did you introduce bias in your study by choosing respondents that were easily accessible – did this mean that would have easy access to internet /survey monkey – did you consider posting the survey out to reduce bias?

The survey was published, but we understand that most of the teachers who agreed were related to centers interested in analyzing the Digital Competence of the teachers and subsequent teachers training programs

Data cleansing has been described. We note your suggestion, being able to add an option that automatically completes with 1 = "very low level", and then lose less data. In any case, the largest number of data lost were duplicated records, in which one of them was not complete. We were able to check it, not just by data matching, but by IP registration.

Round 2

Reviewer 3 Report

Dear authors,

the revision of the paper is well done. I'm very satisfied with your changes.

Please check again your English. I have found some misspellings (like skil with one l and proyects, and a construction with "to studied"). Also, some of the comma placements amazed me. I think, this mainly refers to the new parts in the paper.

Congratulation,

Reviewer

Author Response

ANSWERING THE REVIEW JANUARY 2021:

“Teacher digital literacy: the indisputable challenge after COVID-19”

Dear colleagues,

First of all, we want to thank you for the opportunity to improve the paper. We consider that the latest indications have been, once again, very enriching for the final result of our article.

In the last version of our paper, we have highlighted (in red) all the major modifications made in the text. On the other hand, a revision of the text has been made by a specialist translator on the subject. We have reviewed the format of the text, tables and figures, according to the journal guidelines.

In order to improve the statistical analysis, a regression analysis has been carried out, to obtain a linear multiple regression model, which explains the relationships between the different variables of the questionnaire.

Describing some of the modifications made:

The last lines of the abstract have been modified to include the new statistical analysis and rewritten to fit the maximum number of words.
Image 3 has been presented as a figure and not as a table, because that format was difficult to read and understand. Answer options have been added for questions related to the characteristics of respondents.
The order within the section “Materials and Methods” has been changed, now 2.1 Instrument, 2.2 Population and Sample, 2.3 Method.
In 2.3 Method, the explanation of the descriptive statistical analysis has been expanded, and it has been explained that a multiple linear regression analysis will be performed.
In Figure 4, the names of the 5 integer dimensions have been written.
Figure 5 has been improved.
Figure 6 has been expanded, adding the diagrams for the 5 dimensions of competencies, not just total digital competence. A text is previously added in order to complete the explanation of the figure.
Title of 3.1.2 is expanded to adjust it to the new analysis.
The Pearson's Chi-Squared coefficient value for the 5 dimensions is added in Table 4, and it is indicated if there are dependencies or not. In the previous version only the values for the total digital competence were shown. The explanation of dependency analysis is expanded.
A multiple regression analysis is performed. Adding, Table 5, with the description of the variables used in the model; Table 6, with the coding of the variables; the equations associated with the regression model. In Table 7, the coefficients of the regression model are shown. And a detailed explanation of the model obtained.
In the "Discussion and Conclusions“ section, the paragraphs have been ordered, and some of them modified. In the first paragraph, the global results of the digital competence levels of the sample have been added. The text in which the existing relationships between the variables were discussed has been modified, deepening and adapting it to the new analysis. Some limitations have been added.
Finally, a new bibliographic reference has been added and that section has been reordered.

We hope that all these improvements match all the suggestions done by the reviewers.

Best regards,

The authors

Author Response File: Author Response.pdf