Next Article in Journal
In-Service and Pre-Service Science Teachers’ Enacted Pedagogical Content Knowledge about the Particulate Nature of Matter
Next Article in Special Issue
E-Learning Courses Evaluation on the Basis of Trainees’ Feedback on Open Questions Text Analysis
Previous Article in Journal
Validation of the DigCompEdu Check-in Questionnaire through Structural Equations: A Study at a University in Peru
Previous Article in Special Issue
Effectiveness of Doctoral Defense Preparation Methods
 
 
Article
Peer-Review Record

Categorized and Correlated Multiple-Choice Questions: A Tool for Assessing Comprehensive Physics Knowledge of Students

Educ. Sci. 2022, 12(9), 575; https://doi.org/10.3390/educsci12090575
by Shabnam Siddiqui
Reviewer 1:
Reviewer 2: Anonymous
Reviewer 3: Anonymous
Reviewer 4: Anonymous
Educ. Sci. 2022, 12(9), 575; https://doi.org/10.3390/educsci12090575
Submission received: 20 April 2022 / Revised: 2 August 2022 / Accepted: 15 August 2022 / Published: 23 August 2022
(This article belongs to the Special Issue Assessment and Evaluation in Higher Education—Series 2)

Round 1

Reviewer 1 Report

Creating assessments which measure critical thinking skills is important and would be incredibly valuable to the PER community. The premise of this paper is sound and the overall idea is good. The implementation, however, is not compelling. There is no mention of instrument validation (content validity, construct validity) and there is no explanation for why the development of CCMCQs are providing a better measure than free response questions alone. While it is clear that this type of assessment is superior to pure MQ assessments, it is not clear what benefit an instructor gets from using a CCMCQ assessment over open-ended assessments which requires students to explain their thinking. 

Overall - more work needs to be done in the following broad areas:

1) Better defining the research question - the authors set out to answer the question "is there a better way to design MCQs to improve the learning outcomes in a physics classroom"? This question needs to be better defined. What do we mean by "better" and how are "learning outcomes" measured?

2) The difference between "approach 1" and "approach 2" needs to be much better defined and described. The best I can tell - is approach 1 is the same as approach 2 except approach 2 is analyzing students' responds to CCMCQ questions and approach 1 is analyzing students' responses to MCQs. Is this correct? This needs to be better explained especially with respect to how it answers the research question.  

3) validating CCMCQ's and making it more clear how they are better than FR assessments

4) improving the literature review in the introduction - there are MANY places where claims are made with no citations and no substantiation. In the discussion about the limitations of MCQ, for example (pages 2-3) - many of the statements made seem specific to particular situations and not to MCQs in general 

- another problem with the literature review is - it seems to conflate 2 different bodies of work without a clear explanation of how they are connected/ overlap. On page 5 - the authors say that physics knowledge is described into 3 categories (factual, conceptual, procedural) but then later (on the same page) they cite a different literature that claims physics knowledge actually falls into 3 different categories (conceptual understanding, critical thinking, quantitative understanding). Which of these two frameworks is important for understanding this paper? How do I reconcile the two? Which one is the research question referring to when it mentions measuring learning outcomes? 

5) Use of the word correlation - the authors discuss how CCMCQ's correlate conceptual understanding and quantitative understanding and this is a central point in the paper. However, it is not clear what this means exactly. The word correlates has a statistical meaning but, it seems, that this is not how the authors are using it. This is a major problem. The authors need to explain what they mean by correlated. How do they know questions are correlated? 

6) Table 2 shows the difficulty and point biserials for the 3 items on the 3 exams. In the preceding text, there is some discussion about how a "satisfactory" point biserial is greater or equal to 0.2. Firstly  - no references are included and secondly, there is no explanation about how the point biserials (or difficulty metrics) fit into the research question (partly b/c the research question is really not well defined. 

7) typographical errors/ grammar - there are many words that are weirdly hyphenated (to-wards, ad-vantages, etc); there are also many grammar errors throughout the paper (too many to list here)

8) There are a few places where an acronym is used but not defined until much later in the paper (CCMCQ - for example) - making it very difficult to follow the paper

 

Author Response

Reviewer 1

Creating assessments which measure critical thinking skills is important and would be incredibly valuable to the PER community. The premise of this paper is sound and the overall idea is good. The implementation, however, is not compelling. There is no mention of instrument validation (content validity, construct validity) and there is no explanation for why the development of CCMCQs are providing a better measure than free response questions alone. While it is clear that this type of assessment is superior to pure MQ assessments, it is not clear what benefit an instructor gets from using a CCMCQ assessment over open-ended assessments which requires students to explain their thinking. 

Overall - more work needs to be done in the following broad areas:

Comment 1) Better defining the research question - the authors set out to answer the question "is there a better way to design MCQs to improve the learning outcomes in a physics classroom"? This question needs to be better defined. What do we mean by "better" and how are "learning outcomes" measured?

Response 1) In our manuscript, we first discussed all disadvantages of MCQs based tests (page. 3). All these disadvantages of MCQs described in the paper are supported by peer reviewed references. Later, we asked this question in context with these disadvantages. Here, ‘better means’, a new method, or an alternative method for designing MCQs so that all disadvantages of MCQs can be eliminated by using such a new method.

Improvement: We have formulated this question as, Can there be an alternative method for designing MCQs than existing method of MCQs? Can we design MCQs with a new approach so that the above mentioned disadvantages of MCQs can be overcome? Can this newl method provide improved learning outcomes in a physics classroom?

 

Comment 2) The difference between "approach 1" and "approach 2" needs to be much better defined and described. The best I can tell - is approach 1 is the same as approach 2 except approach 2 is analyzing students' responds to CCMCQ questions and approach 1 is analyzing students' responses to MCQs. Is this correct? This needs to be better explained especially with respect to how it answers the research question.  

Response 2) There is a big difference between the two approaches. In approach 1, all the responses are studied as an independent responses, and there are no correlations (logical connection) between the questions. We have replotted our graph as sets of four questions from three exams. Please see page 16. This new graph shows percentages of incorrect responses of the students for each question. This approach provides information regarding questions that were most difficulty to answer. But it does not provide complete knowledge of the students’ difficulties. It just points out some specific difficulties. For example, the data suggested that the conceptual understanding of physics is most difficult for the students. Also, students somehow can choose a correct equation for solving a problem and are even better at arriving at correct numerical value. The main difference between approach 1 and approach 2 is that approach 1 provides partial information regarding students’ difficulties whereas approach 2 provide complete information regarding students’ difficulties. Approach 2 provide information on students such as, how many students have learned physics concepts and problem solving proficiently, how many are partially proficient in physics, and how many students are just guessing their answers. Such information is impossible to be obtained from standard approach 1. Further, developing question sets using approach 2 and giving such questions as homework and exams will help students to learn to solve problems systematically. This is mainly due to logical connectivity between the questions, and categorization that helps students to identify their difficulties.

Improvement: We have changed our graphs to only independent responses graphs, and removed all categorization such as conceptual, and quantitative responses categorization. This will provide clear distinction between approach 1 and approach 2.

 

Comment 3) validating CCMCQ's and making it more clear how they are better than FR assessments

Response: A detailed table, Table. 1 is provided in the manuscript that describes the differences between MCQs, CCMCQs and FR assessments. We would like to request the reviewer to please take a closer look at this table. This table provide details about the differences between these three methods, advantages and disadvantages. In author’s experience when FR questions are given to the students, most of the time students do not even know where to start to solve the problem. But CCMCQs give students hints to where to start thinking about the question by asking them logically connected questions.

Comment 4) improving the literature review in the introduction - there are MANY places where claims are made with no citations and no substantiation. In the discussion about the limitations of MCQ, for example (pages 2-3) - many of the statements made seem specific to particular situations and not to MCQs in general 

- another problem with the literature review is - it seems to conflate 2 different bodies of work without a clear explanation of how they are connected/ overlap. On page 5 - the authors say that physics knowledge is described into 3 categories (factual, conceptual, procedural) but then later (on the same page) they cite a different literature that claims physics knowledge actually falls into 3 different categories (conceptual understanding, critical thinking, quantitative understanding). Which of these two frameworks is important for understanding this paper? How do I reconcile the two? Which one is the research question referring to when it mentions measuring learning outcomes? 

Response 4) For the first part of the comment the reviewer is not specific about “MANY” places. For the second part of the comment, “On page 2”, we have provided references from [24-29] (a total of five references) that discusses the advantages/disadvantages and issues with MCQs. All the disadvantages with MCQs mentioned in this paper are obtained from these peer reviewed journal papers which are given in references [24-29].

Improvement: for the limitations of MCQs, we have referred references as [24-29] on page 3. This will help the readers to find these references which discusses the disadvantages of the MCQs.

On page 5, we have improved our language to describe what exactly are three categories of physics knowledge in this work. On page 5, physics knowledge is described in three categories from  references [31,40]. Later, the author shows that critical thinking is lacking in a physics classroom. Further, the author then categorizes physics knowledge into three categories as conceptual, critical thinking and quantitative. Here quantitative means the same as procedural.

 

5) Use of the word correlation - the authors discuss how CCMCQ's correlate conceptual understanding and quantitative understanding, and this is a central point in the paper. However, it is not clear what this means exactly. The word correlates has a statistical meaning but, it seems, that this is not how the authors are using it. This is a major problem. The authors need to explain what they mean by correlated. How do they know questions are correlated? 

Response 5) In this paper the word ‘correlate’ means ‘logical connection’ between the questions which author has tried to explain using a problem example on page 7 to page 11.

Improvement: We have added this new information on page. 7.

6) Table 2 shows the difficulty and point biserials for the 3 items on the 3 exams. In the preceding text, there is some discussion about how a "satisfactory" point biserial is greater or equal to 0.2. Firstly  - no references are included and secondly, there is no explanation about how the point biserials (or difficulty metrics) fit into the research question (partly b/c the research question is really not well defined. 

Response: On page 15 of the paper we have provided two references [37, 38] which provides all the information regarding item reliability test, and biserial coefficient. These are standard tests applied to MCQs in physics community.

Improvement: We have made changes to the paper, and elaborated on this point why such measures are used in this manuscript. Please refer to page. 15.

7) typographical errors/ grammar - there are many words that are weirdly hyphenated (to-wards, ad-vantages, etc); there are also many grammar errors throughout the paper (too many to list here).

Response: This is simply a software error. While uploading manuscript on MDPI website there errors arose. The author cannot do anything about them.

8) There are a few places where an acronym is used but not defined until much later in the paper (CCMCQ - for example) - making it very difficult to follow the paper.

Response: The acronym is first defined in the abstract. If reviewer has read the abstract first, then it should not be a problem.

Improvement: We have further added this acronym in the beginning of the manuscript as well.

 

Reviewer 2 Report

Dear authors,

I send you my review report.

Article title: Categorized and correlated multiple-choice questions: A tool for assessing comprehensive physics knowledge of students

 

The authors present in this manuscript a new assessment tool with the acronym “Categorized and Correlated Multiple Choice Questions” (CCMCQs) for evaluating the comprehensive physics knowledge level of students. They discuss the outcomes of a one-semester study on CCMCQs using data obtained from an introductory physics course. The authors have made a significant effort in guiding the reader through the overall study. The problems of this study are detailed below.

Specific comments:

Research manuscripts should comprise Introduction, Materials and Methods, Results, Discussion, and Conclusions. The submitted manuscript did not follow this structure.

ABSTRACT

Nowhere in the abstract or the text of the manuscript is the goal of the presented study clearly stated.

In the abstract the methodology used and the sample from which the data were obtained should be explained.

  1. INTRODUCTION

Before explaining the part of materials and method it is important to make a review of the new scientific literature, following the scientific method. The cited sources are older.

In addition to the mentioned active learning methods, in the introduction, I recommend supplementing, for example, the results of studies the Integrated e-Learning (INTe-L) the new strategy of education.

In the introduction, the authors mention of outcome from their study. I recommend moving this part to the conclusion of the manuscript.

  1. BACKGROUND

Some of the authors' claims concerning MCQs tests are not substantiated by research results.

In the options offered for answering quantitative question 2, the unit meter (m) is not indicated at 21.91.

  1. METHODOLOGY

The meaning of the acronym CCMCQs is explained in the abstract, it is used in the introduction, but its meaning is again clarified in subchapter 3.1.

Question: “Bob throws a ball straight up at 20 m/s, releasing the ball at 1.5m above the ground. Find the maximum height reached by the ball. Assume no air resistance.“

The question is not unequivocally formulated. In my opinion, it should be formulated as follows: “Bob throws a ball straight up at 20 m/s, releasing the ball at 1.5m above the ground. Find the maximum height reached by the ball in consideration of the earth’s surface. Assume no air resistance.“

The value of the velocity at which Bob throws a ball straight up 20 m.s-1 = 72 km.h-1 is not real it is the velocity of the car.

Question 5 states as the correct answer that the final speed at the maximum is a known variable if the answer, therefore, relates to the original question.

Question 6 does not state as a correct answer that the maximum height that the ball reaches is also an unknown variable if the answer, therefore, relates to the original question.

Question 7: “The correct equation for solving the problem is: ...“is not unequivocally formulated. What problem should the equation apply to?

In the options offered for answering question 8, the unit meter (m) is not indicated at 21.91.

In question 9, the data on the axes of the graphs are illegible.

 

RESULTS AND DISCUSSION

There is no discussion of results in the light of scientific knowledge.

Author Response

Reviewer 2

Dear authors,

I send you my review report.

Article title: Categorized and correlated multiple-choice questions: A tool for assessing comprehensive physics knowledge of students

 

The authors present in this manuscript a new assessment tool with the acronym “Categorized and Correlated Multiple Choice Questions” (CCMCQs) for evaluating the comprehensive physics knowledge level of students. They discuss the outcomes of a one-semester study on CCMCQs using data obtained from an introductory physics course. The authors have made a significant effort in guiding the reader through the overall study. The problems of this study are detailed below.

Specific comments:

Research manuscripts should comprise Introduction, Materials and Methods, Results, Discussion, and Conclusions. The submitted manuscript did not follow this structure.

Response: We have presented in this manuscript Introduction, Background, Methodology, Results and Discussion and Conclusions. Since, this is a theoretical study, there are no materials used. We have followed the same procedure as used by other published papers in your journal.

ABSTRACT

Nowhere in the abstract or the text of the manuscript is the goal of the presented study clearly stated.

In the abstract the methodology used and the sample from which the data were obtained should be explained.

Response: The main goal of this paper is to present a new methodology which is very clear in the abstract.

  1. INTRODUCTION

Before explaining the part of materials and method it is important to make a review of the new scientific literature, following the scientific method. The cited sources are older.

In addition to the mentioned active learning methods, in the introduction, I recommend supplementing, for example, the results of studies the Integrated e-Learning (INTe-L) the new strategy of education.

In the introduction, the authors mention of outcome from their study. I recommend moving this part to the conclusion of the manuscript.

Response: We have added journal papers on e-Learning as suggested by the reviewer. Please refer reference [41]

  1. BACKGROUND

Some of the authors' claims concerning MCQs tests are not substantiated by research results.

In the options offered for answering quantitative question 2, the unit meter (m) is not indicated at 21.91.

Response: We have added and checked all units for the answers.

  1. METHODOLOGY

The meaning of the acronym CCMCQs is explained in the abstract, it is used in the introduction, but its meaning is again clarified in subchapter 3.1.

Response: We have moved it to beginning of the paper.

Question: “Bob throws a ball straight up at 20 m/s, releasing the ball at 1.5m above the ground. Find the maximum height reached by the ball. Assume no air resistance.“

The question is not unequivocally formulated. In my opinion, it should be formulated as follows: “Bob throws a ball straight up at 20 m/s, releasing the ball at 1.5m above the ground. Find the maximum height reached by the ball in consideration of the earth’s surface. Assume no air resistance.“

Response: This question is taken from a reference [31]. It is a standard practice in physics courses to call ‘ground’ as ‘earth’s surface’. We do not see any reason to change this language.

The value of the velocity at which Bob throws a ball straight up 20 m.s-1 = 72 km.h-1 is not real it is the velocity of the car.

Question 5 states as the correct answer that the final speed at the maximum is a known variable if the answer, therefore, relates to the original question.

Response: This question is a standard question from physics textbook. The main goal of such questions is not to present velocities of real scenarios. But to give problems with these numbers in SI units (m/s). At the same time, one does not wish to make such numbers very small such as 1m/s or 2 m/s.

The final speed is always a known variable at maximum height because it is taught in physics classes when an object is thrown up, its velocity keeps decreasing and becomes zero at the maximum height. But final velocity when the object lands on the ground is not a known variable. The reviewer is getting confused between the final velocity at maximum height and final velocity when object reaches the ground.

Question 6 does not state as a correct answer that the maximum height that the ball reaches is also an unknown variable if the answer, therefore, relates to the original question.

Response: We do not understand the reviewers comment here.

Question 7: “The correct equation for solving the problem is: ...“is not unequivocally formulated. What problem should the equation apply to?

Response: We do not understand the reviewers’ questions here.

In the options offered for answering question 8, the unit meter (m) is not indicated at 21.91.

Response: We have added unit. It was a typing mistake.

In question 9, the data on the axes of the graphs are illegible.

Response: We do not understand, how the data is illegible. We have checked one more time and made our graphs more readable.

Reviewer 3 Report

In the paper "Categorized and correlated multiple-choice questions: A tool for assessing comprehensive physics knowledge of students" the authors present a test developed by them to assess conceptual understanding in the field of mechanics. They claim that their test is superior to classical multiple-choice questionnaires. The multiple-choice questionnaires they compare their test to are e.g. the Mechanics Baseline Test or the Force Concept Inventory (FCI). These tests have been extensively validated on the basis of empirical data (e.g. content validity, cognitive validity, structural validity, external validity), whereas the authors of this paper do not provide any real empirical evidence that their test is better with respect to the different validity aspects. They only show that the student answers can be evaluated in a more differentiated way. In order to strengthen the statement of the article that the new test is superior to other test procedures, either a concrete focus would have to be placed on a validity aspect (e.g. structural validity by measuring the reliability of the test) or a overall validity argumentation according to e.g. Kane would have to be pursued. Overall, a concrete empirical research goal or research question with the focus on validity is missing.

Which concept of reliability is referred to on page 15. Because here are some inconsistencies in the presentation, if one refers to Cronbach's Alpha: What is individual item reliability? A scale has a reliability not an individual item. Item difficulty does not directly affect reliability. Difficult items can also contribute to a reliable scale. But only if they are not randomly answered correctly or incorrectly to the same extent by high-performing and low-performing students.

Even if a differentiated assessment of conceptual understanding is a reasonable goal for test development and maybe the developed test can achieve this, this paper has too many empirical deficiencies. 

Author Response

Reviewer 3

Comments and Suggestions for Authors

In the paper "Categorized and correlated multiple-choice questions: A tool for assessing comprehensive physics knowledge of students" the authors present a test developed by them to assess conceptual understanding in the field of mechanics. They claim that their test is superior to classical multiple-choice questionnaires. The multiple-choice questionnaires they compare their test to are e.g. the Mechanics Baseline Test or the Force Concept Inventory (FCI). These tests have been extensively validated on the basis of empirical data (e.g. content validity, cognitive validity, structural validity, external validity), whereas the authors of this paper do not provide any real empirical evidence that their test is better with respect to the different validity aspects. They only show that the student answers can be evaluated in a more differentiated way. In order to strengthen the statement of the article that the new test is superior to other test procedures, either a concrete focus would have to be placed on a validity aspect (e.g. structural validity by measuring the reliability of the test) or a overall validity argumentation according to e.g. Kane would have to be pursued. Overall, a concrete empirical research goal or research question with the focus on validity is missing.

Which concept of reliability is referred to on page 15. Because here are some inconsistencies in the presentation, if one refers to Cronbach's Alpha: What is individual item reliability? A scale has a reliability not an individual item. Item difficulty does not directly affect reliability. Difficult items can also contribute to a reliable scale. But only if they are not randomly answered correctly or incorrectly to the same extent by high-performing and low-performing students.

Even if a differentiated assessment of conceptual understanding is a reasonable goal for test development and maybe the developed test can achieve this, this paper has too many empirical deficiencies. 

Response: In this work, we have discussed that MCQ’s based tests have several disadvantages. These disadvantages have been discussed in several peer reviewed journal papers. The Mechanics Baseline Test or the Force Concept Inventory have been extensively validated. But this does not imply that these tests are unflawed in their approach, and nothing new is needed to make improvements. There are several journal papers in the literature that have discussed the disadvantages of these tests. Even though FCI test has been implemented at various Universities, it still can be further improved. In this paper we have tried to suggest an improvement for conducting MCQs based tests. We are not saying that FCI are not good tests. We are only trying to convey that improvements can be made by using this new technique. Using CCMCQs, one can perform a better evaluation of students learning and also help them to learn physics concepts and problem solving more systematically. Which is almost impossible with the standard tests.

For reliability test, we have added a paragraph to explain the main objective of this paper for performing such tests. We have mentioned references [37-38] and have used these references for our work on item analysis.  We would like to suggest the reviewer to read these references for item reliability.

 

 

Reviewer 4 Report

Dear authors,

The manuscript describes an interesting work on physics knowledge assessment. The authors present their case in a thorough manner.

However, despite the fruitful quantitative analysis of the data, other sources of data will empower their argument concerning the reliability of the approach. Since “This work is a preliminary step towards the development of this methodology”, I propose to the authors explain their decision not to collect and analyze data (in a systematic way) for their students using another (probably qualitative) method to ensure the reliability of the method and validity of the results. That could be included in a section at end of the manuscript called “Limitations and future studies” (I have seen the lines 657-675, but in these lines, you just describe your impression and thoughts).

Minor comment:

571: The graph is not clear. Please improve the quality

 

 

Regards,

Author Response

Reviewer 4

Dear authors,

The manuscript describes an interesting work on physics knowledge assessment. The authors present their case in a thorough manner.

However, despite the fruitful quantitative analysis of the data, other sources of data will empower their argument concerning the reliability of the approach. Since “This work is a preliminary step towards the development of this methodology”, I propose to the authors explain their decision not to collect and analyze data (in a systematic way) for their students using another (probably qualitative) method to ensure the reliability of the method and validity of the results. That could be included in a section at end of the manuscript called “Limitations and future studies” (I have seen the lines 657-675, but in these lines, you just describe your impression and thoughts).

Minor comment:

571: The graph is not clear. Please improve the quality

 

 Response: We have added a new section, Section 6 for Limitations and future studies. We have improved the quality of the grapph. 

 

Round 2

Reviewer 1 Report

I feel as though the edits made to this paper adequately address the concerns I previously expressed. 

Author Response

I appreciate all your inputs for the manuscript. Thank you!

Reviewer 2 Report

Dear authors,

congratulations on improving your manuscript. You have improved the clarity of your writing. The revision clarifies almost all the points I raised.  

Kind regards,

The Reviewer

Author Response

I appreciate all your inputs for the manuscript. Thank you!

Reviewer 3 Report

In reply to your comments to my review: I’m not questioning that the FCI has disadvantages and it’s good to improve MCQs based test, I’m questioning, that you do not provide empirical evidence, that your test is valide and has more advantages than disadvantages than existing ones in relation to common test quality criteria. This means for example from a qualitative perspective a comparison of students answers on a common MCQs based test and your developed test. Or on a quantitative perspective a comparison of test quality criteria between those types of tests (reliability, structural validity, etc.).  

You have mentioned two references, which are your basis of your statistical analyses [37, 38].. I now understand your approach for the analysis of the quality of an individual item. What's missing is an analysis of the reliability of the test in whole. For example in relation to one of your references the “Kuder-Richardson reliability index [is such a measure]. These [….] measure [… is] used to evaluate an entire test rather than evaluate individual items. Kuder-Richardson reliability measures the internal consistency of a test. In other words, it examines whether or not a test is constructed of parallel items that address the same materials…” [37, p.2] So you gave empirical hints that every item discriminates between students abilities, but you didn’t gave empirical hints that the three items measure the same construct. So, you can’t say that your test measures content knowledge as a construct. Where as individual item quality measures are necessary but not sufficient to state that your test as a whole has a good internal structural validity).  So you picked out only the individual quality criteria of an item, but not the criteria which measure the quality of the test in whole. Although these are the common quantitative quality criteria of a test, which can be compared between different tests. 

You have not optimized your contribution in relation to my two criticism, so I stand by my previous assessment of your manuscript.

Author Response

Response: In this research, the focus was to test the reliability of each item designed using CCMCQs. The whole exam was not constructed using CCMCQs and only a section of the exam was constructed using CCMCQs questions. This work is a first step towards gaining an understanding of the advantages/disadvantages of CCMCQs. Constructing a whole exam based on CCMCQs would be our second step and we plan to conduct such studies in the future. For our first step, we have demonstrated that by constructing MCQs using CCMCQs approach, more valuable information can be gained from the data. We have shown reliability of the four CCMCQs questions (three items) posed in the exam. The measure of reliability of each item is called as point biserial coefficient. The point biserial coefficient is a measure of individual item reliability and is defined as the correlation between the item scores and total scores (a total score of four questions).

We did not design the whole exam on CCMCQs, only a section of the exam was constructed on CCMCQs. We called such a section, a test (a set of four questions). We used the word ‘test’ to separate it from the word ‘exam’. Exam means all the questions in all the sections. We never claimed that whole exam was designed on CCMCQs, and that is why we did not use Kuder-Richardson reliability measure. This measure is not needed for our research at this point.  In order to test the reliability of the whole exam we need to design the whole exam based on CCMCQs. The three items that we have used does not measure the same construct. In our paper, by test we mean a set of four CCMCQs questions. So, there were a total of three sets of CCMCQs for the three exams. We have presented this information in the graphs as well in the text. We guess that reviewer got confused between the word test and exam. We have added the meaning of ‘test’ in our paper to avoid such confusion. In this work we have demonstrated that using CCMCQs comprehensive knowledge of the students (on three different topics) can be evaluated.

 

Reviewer 4 Report

Dear authors, 

thank you for taking into consideration my suggestion. 

Good luck.

Author Response

I appreciate all your inputs for the manuscript. Thank you!

Back to TopTop