Attitudes towards Research Methods in Education: Development of the ATRMQ Scale

Round 1

Reviewer 1 Report

This study reports on the adaptation and validation of a questionnaire for undergraduate students studying education, which investigates their attitudes towards research methodology. The topic is not only relevant in the Spanish context, as evidence-based decision-making is playing an increasing role in both educational theory and practice. To improve the manuscript, I have a few suggestions.

- In the Introduction, the author(s) justify the research by stating that “However, in reviewing the literature, it is noteworthy that, although there are studies on Education students' attitudes towards Statistics (see [17]), there are no studies related to RM in Education.” (lines 60-62). There is a large body of research in the literature on teacher candidates' attitudes towards research and research-based teaching. It would be important to review and summarise these and reflect on them to show the novelty and importance of the current research.

- The theoretical background of the attitude survey is also rather modest, and mainly concerns the assessment of the attitudes of psychology and education students towards statistics.

- The authors used a questionnaire on attitudes towards statistics, replacing the term statistics with research method. Whether this provides enough information, since the topic to be studied, the research methodology courses, is much more complex.

- Another problem, which the author(s) also pointed out among the limitations, is that they cannot be sure what the students meant by "research methods" and what subjects they meant when they answered.

- The authors do not explain why they chose this CAHE questionnaire. The structure of the questionnaire is inconsistent with the three-component model mentioned in the theoretical background, which emphasises the cognitive, affective, and social aspects of attitude.

- It is disturbing that in the case of both the original questionnaire (CAHE) and its revised version (CAHMI) the author(s) use the abbreviation of the Spanish name. The meaning of the abbreviations is not given in the summary or the manuscript. It would be important to give the English name of the questionnaires and to form an abbreviation based on the English name.

- I think it would have been more appropriate to use the term "research methodology" in the questionnaire instead of "research methods". This would have more precisely defined the area to be investigated and the related courses.

- Why was random sampling used?

- The sample is heterogeneous by degree, age, and grade. There is no information on the form and depth of research methodology studied by undergraduate students in each degree programme.

- It would also be useful to check the validity of the questionnaire for subsamples of different educational degrees.

- All values of the Likert scale should be given (line 126).

- “A sixth option was added for respondents who did not know how to answer (6= I don't know) to differentiate lack of knowledge from a neutral position.” (lines 126-127). Why was this necessary? Were there students in the sample who had not studied research methodology?

- Those who selected option 6 were excluded from the data analysis. What was this proportion?

- Pallant (2010) is missing from the references.

- There is no justification as to why the database was split into two parts for the EFA and the CFA. On what basis were the data divided?

- In Table 1, only factor loadings above 0.4 should be displayed.

- In the Discussion, the author(s) describe that the three-dimensional structure confirmed by factor analysis is consistent with the literature (lines 265-266). However, due to the characteristics of this questionnaire, the affective and cognitive factors are presented here, while the behavioural one is missing.

- The findings on attitudes towards RM content as a construct are influenced by the nature of the present survey, which should be taken into account when concluding the theory of attitude research.

Author Response

Dear Reviewer,

We greatly appreciate your insightful comments on our manuscript and have carefully addressed each point in the attached PDF. Your feedback has been invaluable in enhancing our work, ensuring it meets the journal's standards of excellence. We remain open to further suggestions and look forward to your continued guidance.

Author Response File: Author Response.pdf

Reviewer 2 Report

Dear Authors,

I hope this message finds you well. I have carefully reviewed your manuscript, titled " Attitudes Towards Research Methods in Education: Development of the CAHMI Scale". Your work addresses an important topic; however, several key areas need attention to strengthen the manuscript before publication:

·         Outdated theoretical framework: Please update your theoretical framework with recent references from the last five years. This will ensure the study's theoretical foundation is current and relevant to the field.

·         Refinement of theoretical framework: The theoretical framework provided is quite broad. I recommend narrowing it down to clearly articulate the research problem and emphasize the importance of validating the scale within a specific context.

·         Relevance of the study: Strengthen the justification for understanding students' attitudes toward research methods content. Highlight the significance of your findings for educators and suggest practical applications based on your results.

·         Low Cronbach's Alpha in utility dimension: Address the reported low Cronbach's alpha in the utility dimension. Identify which items are contributing to this issue and thoroughly analyse their impact. Provide insights into potential improvements in this area.

·         Scale improvement: Since you acknowledge the need for improvement in linking factors to positions against or in favour, delve deeper into this issue. Provide a more detailed analysis and propose potential solutions for enhancing the scale.

·         Ethical considerations: It is crucial to include information on the ethical treatment of your research. Provide details on whether the study underwent ethical review by a committee and how ethical considerations were managed throughout the research process.

·         Expanded conclusion: Your conclusions are concise and lack specificity. Please expand on your findings, discussing their implications and potential applications in educational settings.

I believe addressing these points will significantly improve the quality and relevance of your manuscript.

Best regards,

Author Response

Dear Reviewer,

We greatly appreciate your insightful comments on our manuscript and have carefully addressed each point in the attached PDF. Your feedback has been invaluable in enhancing our work, ensuring it meets the journal's standards of excellence. We remain open to further suggestions and look forward to your continued guidance.

Author Response File: Author Response.pdf

Reviewer 3 Report

Thank you for submitting your paper, "Attitudes Towards Research Methods in Education: Development of the CAHMI Scale," to the Education Sciences journal. It is a pleasure to review your work.

I agree with you that having a solid understanding of research knowledge is essential for teachers, particularly in applying these methodologies to practice. This necessity underscores the importance of instilling such knowledge in student teachers as early as their university education to prepare them adequately for their future roles as educators.

Your emphasis on the significant role that attitudes play in this context is well-founded. Attitudes toward research methods can significantly influence both the learning process and the application of these methods in educational settings. Developing positive attitudes towards research methods early in their education can equip student teachers with the confidence and skills needed to incorporate evidence-based practices into their teaching.

Nevertheless, I would like to point out some clarifications and potential enhancements:

1. The statement, "however, in reviewing the literature, it is noteworthy that although there are studies on Education students' attitudes towards Statistics (see [17]), there are no studies related to RM in Education"​​, seems to overlook existing research that may not be entirely similar but worth acknowledging. Examples:

Student attitudes toward research in an undergraduate social science research methods course (DOI: 10.1080/23752696.2022.2072362)

Consuming, producing, and justifying: Finnish student teachers’ views of research methods (DOI: 10.1080/1743727X.2020.1737003)

To address this, I recommend conducting a more comprehensive literature review to identify studies that could provide insights into related areas. You MAY also consider studies that examine attitudes toward research in general (and not just research methods), and analyses that recruit students from other disciplines (and not just student teachers). By acknowledging these studies, you can provide a richer context for your research, demonstrating awareness of the broader academic landscape. This approach will not only refine the justification for your study but also enhance its contribution to the field by situating it within a more detailed scholarly background.

2. Your approach to incorporating Statistics-related instruments in your study is insightful. However, considering the specific emphasis of your study on research methods in education, it may be beneficial to broaden the scope of instruments reviewed or included in your analysis to encompass those more directly related to research. This adjustment could provide a more nuanced understanding of the attitudes toward research methods among education students. Examples:

Psychometric Properties and Factor Structure of the Attitudes Toward Research Scale in a Graduate Student Sample (DOI: 10.1177/1475725719842695)

Scale of Attitudes Towards Research (EACIN-R): Psychometric Properties in Peruvian University Students (10.20511/pyr2023.v11n1.1699)

Attitudes Toward Research Scale--30 item (10.1037/t35505-000)

Note: I did not write any of these papers. I am simply pointing out that there are similar research that exist already.

3. With this in mind, it is imperative to articulate a clear rationale for selecting the Attitudes toward Statistics Questionnaire (CAHE) as a primary basis for your adapted instrument. This justification should address the questionnaire's relevance and effectiveness in capturing aspects of attitudes towards research methods, despite its primary focus on statistics.

You might consider elaborating on the parallels between statistical understanding and research methods competencies, highlighting how the CAHE might indirectly measure or reflect upon attitudes toward research methods. Additionally, discussing any modifications made to the CAHE or supplementary methods employed to ensure it adequately captures attitudes towards research methods could strengthen your methodology section.

4. The adaptation of the CAHE to create the CAHMI by merely changing "statistics" to "research methods" raises questions about the instrument's validity for assessing attitudes towards research methods specifically. Although a group of education teachers evaluated the semantic and grammatical consistency of the items, a more rigorous validation process might be necessary. This could include pilot testing the adapted instrument with a smaller, diverse sample of education students to gather preliminary data on its reliability and validity in the new context. Additionally, considering the development of entirely new items or dimensions that capture unique aspects of attitudes toward research methods could enhance the instrument's relevance and specificity.

5. By the way, is CAHMI supposed to be an abbreviation? If yes, it's essential to clearly define it within your paper. A full explanation of what each letter stands for, especially in the context of your study, would significantly enhance clarity. This explanation should ideally be placed at the first instance where CAHMI is mentioned, accompanied by a brief description of the instrument's purpose and relevance to your research on attitudes toward research methods in education.

6. The sample comprises a significant majority of female participants (87.9%) and lacks first-year students. While the exclusion of first-year students is justified to ensure familiarity with Research Methodology subjects, considering the gender imbalance and broader diversity in academic backgrounds could strengthen the generalizability of your findings. Exploring attitudes across different academic years and balancing the gender ratio could provide more nuanced insights into how attitudes toward research methods evolve throughout university education.

7. While the methodology section details the psychometric analyses conducted, including exploratory and confirmatory factor analyses, further justification for the choice of these specific methods and their implementation would be valuable. For instance, detailing the rationale behind the choice of factor extraction and rotation methods in the EFA, or discussing the decision-making process for model selection in the CFA, could provide readers with deeper insights into the analytical rigor of the study.

8. While the discussion connects the findings to the broader context of research methods, expanding on how these attitudes might impact future educational practices and policy could offer greater depth. I suggest discussing potential implications for curriculum design, teacher training programs, and student support services would provide readers with a clearer understanding of the study's relevance.

9. Another suggestion that I have is to elaborate on practical applications of the CAHMI scale. For instance, discussing how educators, administrators, and policymakers can use the scale to identify areas for intervention or to measure the effectiveness of educational innovations could make the findings more actionable.

Although I am rejecting the paper in its current form, I am open to reviewing the revised version. Thank you!

Minor editing of English language required

Author Response

We've carefully considered and implemented the modifications suggested by you and other reviewers, significantly enhancing our manuscript. A detailed account of these changes and our responses is provided in the attached document. We believe these revisions have greatly improved the quality of our work, aligning it more closely with the journal's standards. Thank you for your valuable feedback.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

The authors have responded in detail to the comments and the manuscript has been corrected accordingly. The additions have clarified the context and limitations of the research and provided clear direction for further research.

I just have one more suggestion. Since the name CAHE may be known in the Spanish literature, it would be worth mentioning the Spanish name and abbreviation of the questionnaire in lines 114-116 or, if possible, in a footnote (first place in the abstract), and indicating that the English name is used in the study.

Author Response

We sincerely appreciate your continued engagement and the constructive suggestion provided in this second phase of review. Your insight into the importance of acknowledging the original Spanish name and abbreviation of the questionnaire (CAHE) has been invaluable. In response to your recommendation, we have incorporated a clarification in line 140, stating "(named CAHE in its original version in Spanish)" to ensure readers are aware of the questionnaire's origins and its established recognition within the Spanish literature. This addition aims to maintain the integrity and continuity of academic discourse across languages while ensuring the study's accessibility and relevance to an international audience. We are grateful for your thoughtful guidance, which has undoubtedly enriched the manuscript and its connection to the broader body of research. Thank you once again for your valuable contribution to refining our work.

Reviewer 3 Report

Thank you for revising your paper. I am now accepting this version! Congratulations to the authors.

Minor editing of English language required

Author Response

We are pleased to inform you that a third revision has been undertaken to address the minor typographical errors and language use noted in your review. These final adjustments ensure the manuscript meets the linguistic precision expected in scholarly publications.

Thank you for your meticulous attention to detail, which has greatly contributed to the refinement of our paper.