The Reliability and Medical Students’ Appreciation of Certainty-Based Marking
Abstract
:1. Introduction
2. Method
2.1. Study Design
2.2. Data Collection
2.3. Sample
2.4. Data Analysis
3. Results
3.1. Participation
3.2. Student Previous Experience
3.3. Students’ Feedback
3.4. Reliability of CBM in Terms of Learner Types and Students’ Self-Reported Study Efforts
4. Discussion
4.1. Students’ Previous Experience
4.2. Students’ Feedback
4.3. Reliability of CBM in Terms of Learner Types and Students’ Self-Reported Study Efforts
5. Conclusions
5.1. Usefulness of CBM
5.2. Limitations
Supplementary Materials
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
Abbreviations
CBM | Certainty-Based Marking |
MCQ | Multiple choice question |
References
- Coombs, C.; Milholland, J.E.; Womer, F.B. The assessment of partial knowledge. Educ. Psychol. Meas. 1956, 16, 13–37. [Google Scholar] [CrossRef]
- Hevner, K. A method of correcting for guessing in true-false tests and empirical evidence in support of it. J. Soc. Psychol. 1932, 3, 359–362. [Google Scholar] [CrossRef]
- Soderquist, H.O. A New Method Weight. Scores A True-False Test. J. Educ. Res. 1936, 30, 290–292. [Google Scholar] [CrossRef]
- Gardner-Medwin, T. Optimisation of Certainty-Based Assessment Scores. 2013. Available online: https://tmedwin.net/~ucgbarg/tea/IUPS_2013a.pdf (accessed on 10 November 2021).
- Ahlgren, A. Reliability, Predictive Validity, and Personality Bias of Confidence Weighted Scores. 1969. Available online: https://www.ucl.ac.uk/lapt/ahlgren.pdf (accessed on 10 November 2021).
- Gardner-Medwin, A. Certainty-Based Marking: Rewarding Good Judgment of What Is or Is Not Reliable. 2008. Available online: https://www.tmedwin.net/~ucgbarg/tea/innovation2008.html (accessed on 10 November 2021).
- Luetsch, K.; Burrows, J. Certainty rating in pre-and post-tests of study modules in an online clinical pharmacy course—A pilot study to evaluate teaching and learning. BMC Med. Educ. 2016, 16, 267. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Barr, D.A.; Burke, J.R. Using confidence-based marking in a laboratory setting: A tool for student self-assessment and learning. J. Chiropr. Educ. 2013, 27, 21–26. [Google Scholar] [CrossRef]
- Hendriks, W.J.A.J.; Bakker, N.; Pluk, H.; de Brouwer, A.; Wieringa, B.; Cambi, A.; Zegers, M.; Wansink, D.G.; Leunissen, R.; Klaren, P.H.M. Certainty-based marking in a formative assessment improves student course appreciation but not summative examination scores. BMC Med. Educ. 2019, 19, 178. [Google Scholar] [CrossRef] [Green Version]
- Hunt, D.P. The concept of knowledge and how to measure it. J. Intellect. Cap. 2003, 4, 100–113. [Google Scholar] [CrossRef] [Green Version]
- Agrawal, S.; Norman, G.R.; Eva, K.W. Influences on medical students’ self-regulated learning after test completion. Med. Educ. 2012, 46, 326–335. [Google Scholar] [CrossRef]
- Ende, J. Feedback in Clinical Medical Education. JAMA 1983, 250, 777–781. [Google Scholar] [CrossRef]
- Bienstock, J.L.; Katz, N.T.; Cox, S.M.; Hueppchen, N.; Erickson, S.; Puscheck, E.E. To the point: Medical education reviews—Providing feedback. Am. J. Obstet. Gynecol. 2007, 196, 508–513. [Google Scholar] [CrossRef] [PubMed]
- Bing-You, R.; Hayes, V.; Varaklis, K.; Trowbridge, R.; Kemp, H.; McKelvy, D. Feedback for Learners in Medical Education: What Is Known? A Scoping Review. Acad. Med. 2017, 92, 1346–1354. [Google Scholar] [CrossRef] [PubMed]
- Wigton, R.S.; Patil, K.D.; Hoellerich, V.L. The effect of feedback in learning clinical diagnosis. J. Med. Educ. 1986, 61, 816–822. [Google Scholar] [CrossRef]
- Moodle. 2020. Available online: https://docs.moodle.org/311/en/Using_certainty-based_marking (accessed on 1 September 2021).
- Larsen, D.P.; Butler, A.C.; Roediger, H.L., III. Test-enhanced learning in medical education. Med. Educ. 2008, 42, 959–966. [Google Scholar] [CrossRef]
- Colthart, I.; Bagnall, G.; Evans, A.; Allbutt, H.; Haig, A.; Illing, J.; McKinstry, B. The effectiveness of self-assessment on the identification of learner needs, learner activity, and impact on clinical practice: BEME Guide no. 10. Med. Teach. 2008, 30, 124–145. [Google Scholar] [CrossRef] [PubMed]
- Clark, C. The Impact of Confidence-Based Marking on Unit Exam Achievement in a High School Physical Science Course. In Graduate Research Papers; University of Northern Iowa: Cedar Falls, IA, USA, 2020. [Google Scholar]
- Nožica, B.; Čarapina, M.; Staničić, O. Certainty based marking: The use of modified grading schema in multiple choice quizzes. In Proceedings of the 6th International Conference of Education, Research and Innovation, Seville, Spain, 18–20 November 2013; pp. 2160–2165. [Google Scholar]
- Cisar, S.M.; Cisar, P.; Pinter, R. True/false questions analysis using computerized Certainty-Based Marking tests. In Proceedings of the 2009 7th International Symposium on Intelligent Systems and Informatics, Subotica, Serbia, 25–26 September 2009; pp. 171–174. [Google Scholar] [CrossRef]
- Wu, Q.; Vanerum, M.; Agten, A.; Christiansen, A.; Vandenabeele, F.; Rigo, J.M.; Janssen, R. Certainty-Based Marking on Multiple-Choice Items: Psychometrics Meets Decision Theory. Psychometrika 2021, 86, 518–543. [Google Scholar] [CrossRef] [PubMed]
- Cook, D.A.; Levinson, A.J.; Garside, S.; Dupras, D.M.; Erwin, P.J.; Montori, V.M. Internet-based learning in the health professions: A meta-analysis. JAMA 2008, 300, 1181–1196. [Google Scholar] [CrossRef]
- Burton, R.F. Quantifying the Effects of Chance in Multiple Choice and True/False Tests: Question selection and guessing of answers. Assess. Eval. High. Educ. 2001, 26, 41–50. [Google Scholar] [CrossRef]
- Frary, R.B. Formula Scoring of Multiple-Choice Tests (Correction for Guessing). Educ. Meas. Issues Pract. 1988, 7, 33–38. [Google Scholar] [CrossRef]
- Lesage, E.; Valcke, M.; Sabbe, E. Scoring methods for multiple choice assessment in higher education—Is it still a matter of number right scoring or negative marking? Stud. Educ. Eval. 2013, 39, 188–193. [Google Scholar] [CrossRef]
- Wilkinson, K.; Dafoulas, G.; Garelick, H.; Huyck, C. Are quiz-games an effective revision tool in Anatomical Sciences for Higher Education and what do students think of them? Br. J. Educ. Technol. 2020, 51, 761–777. [Google Scholar] [CrossRef]
- Wu, Q.; De Laet, T.; Janssen, R. Elimination Scoring Versus Correction for Guessing: A Simulation Study. In Quantitative Psychology; Wiberg, M., Culpepper, S., Janssen, R., González, J., Molenaar, D., Eds.; Springer International Publishing: Cham, Switzerland, 2018; pp. 183–193. [Google Scholar]
- Bradbard, D.A.; Parker, D.F.; Stone, G.L. An Alternate Multiple-Choice Scoring Procedure in a Macroeconomics Course. Decis. Sci. J. Innov. Educ. 2004, 2, 11–26. [Google Scholar] [CrossRef]
- Gardner-Medwin, T.; Curtin, N. Certainty-Based Marking (CBM) for Reflective Learning and Proper Knowledge Assessment. 2007. Available online: https://www.reap.ac.uk/reap/reap07/Portals/2/CSL/t2%20-%20great%20designs%20for%20assessment/raising%20students%20meta-cognition/Certainty_based_marking_for_reflective_learning_and_knowledge_assessment.pdf (accessed on 1 September 2021).
- Hassmén, P.; Hunt, D.P. Human Self-Assessment in Multiple-Choice Testing. J. Educ. Meas. 1994, 31, 149–160. [Google Scholar] [CrossRef]
- Leclercq, D. Confidence marking: Its use in testing. Eval. Educ. 1982, 6, 161–287. [Google Scholar] [CrossRef] [Green Version]
- Coffield, F.; Moseley, D.; Hall, E.; Ecclestone, K. Learning Styles and Pedagogy in Post-16 Learning: A Systematic and Critical Review; Learning and Skills Research Centre: London, UK, 2004. [Google Scholar]
- Fleming, N.D.; Mills, C. Not Another Inventory, Rather a Catalyst for Reflection. Improv. Acad. 1992, 11, 137–155. [Google Scholar] [CrossRef] [Green Version]
- Hernandez, J.; Vasan, N. Learning styles among medical students: Kinesthetic learners approach to learning anatomy. FASEB J. 2018, 32, 508. [Google Scholar] [CrossRef]
- Husmann, P.R.; O’Loughlin, V.D. Another Nail in the Coffin for Learning Styles? Disparities among Undergraduate Anatomy Students’ Study Strategies, Class Performance, and Reported VARK Learning Styles. Am. Assoc. Anat. 2019, 12, 6–19. [Google Scholar] [CrossRef] [PubMed]
- Hernandez, J.E.; Vasan, N.; Huff, S.; Melovitz-Vasan, C. Learning Styles/Preferences Among Medical Students: Kinesthetic Learner’s Multimodal Approach to Learning Anatomy. Med. Sci. Educ. 2020, 30, 1633–1638. [Google Scholar] [CrossRef]
- Pashler, H.; McDaniel, M.; Rohrer, D.; Bjork, R. Learning Styles: Concepts and Evidence. Psychol. Sci. Public Interest 2008, 9, 105–119. [Google Scholar] [CrossRef] [Green Version]
- Holley, J.H.; Jenkins, E.K. The Relationship Between Student Learning Style and Performance on Various Test Question Formats. J. Educ. Bus. 1993, 68, 301–308. [Google Scholar] [CrossRef]
- Price, P.C.; Jhangiani, R.; Chiang, I.C.A.; Leighton, D.C.; Cuttler, C. Setting Up a Factorial Experiment. Available online: https://opentext.wsu.edu/carriecuttler/chapter/9-1-setting-up-a-factorial-experiment/ (accessed on 10 November 2021).
- Light, G.; Calkins, S.; Luna, M.; Drane, D. Assessing the Impact of a Year-Long Faculty Development Program on Faculty Approaches to Teaching. Int. J. Teach. Learn. High. Educ. 2009, 20, 168–181. [Google Scholar]
- Creswell, J.W. Planning, Conducting, and Evaluating Quantitative and Qualitative Research; Pearson: Boston, MA, USA, 2012; pp. 534–576. [Google Scholar]
- Hadi, P.; Jamil, S.; Javaher, K.; Masood, Y.; Mohammad, R.; Yasemi, M.Y.; Nahal, M.M.; Karim, H. Using VARK Approach for Assessing Preferred Learning Styles of First Year Medical Sciences Students: A Survey from Iran. J. Clin. Diagn. Res. 2014, 8, GC01–GC04. [Google Scholar] [CrossRef]
- Limited, V.L. The VARK Questionnaire (Version 8.01). 2019. Available online: https://vark-learn.com/the-vark-questionnaire/ (accessed on 2 March 2020).
- Tsang, S.; Royse, C.; Terkawi, A. Guidelines for developing, translating, and validating a questionnaire in perioperative and pain medicine. Saudi J. Anaesth. 2017, 11, S80–S89. [Google Scholar] [CrossRef] [PubMed]
- Mukhtar, K.; Javed, K.; Arooj, M.; Sethi, A. Advantages, Limitations and Recommendations for online learning during COVID-19 pandemic era. Pak. J. Med. Sci. 2020, 36, S27–S31. [Google Scholar] [CrossRef] [PubMed]
- Aguilera-Hermida, A.P. College students’ use and acceptance of emergency online learning due to COVID-19. Int. J. Educ. Res. Open 2020, 1, 100011. [Google Scholar] [CrossRef]
- Chatterjee, I.; Chakraborty, P. Use of Information Communication Technology by Medical Educators Amid COVID-19 Pandemic and Beyond. J. Educ. Technol. Syst. 2021, 49, 310–324. [Google Scholar] [CrossRef]
- Schoendorfer, N.; Emmett, D. Use of certainty-based marking in a second-year medical student cohort: A pilot study. Adv. Med. Educ. Pract. 2012, 3, 139–143. [Google Scholar] [CrossRef] [Green Version]
- Almendingen, K.; Morseth, M.S.; Gjølstad, E.; Brevik, A.; Tørris, C. Student’s experiences with online teaching following COVID-19 lockdown: A mixed methods explorative study. PLoS ONE 2021, 16, e0250378. [Google Scholar] [CrossRef]
Independent Variable | p-Value | ||
---|---|---|---|
Accuracy | CBM Bonus | Certainty-Based Score | |
Learner type | 0.900 | 0.711 | 0.932 |
Learning start | 0.512 | 0.940 | 0.453 |
Rehearsal times | 0.593 | 0.862 | 0.423 |
Self-evaluated knowledge | 0.901 | 0.492 | 0.718 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Smrkolj, Š.; Bančov, E.; Smrkolj, V. The Reliability and Medical Students’ Appreciation of Certainty-Based Marking. Int. J. Environ. Res. Public Health 2022, 19, 1706. https://doi.org/10.3390/ijerph19031706
Smrkolj Š, Bančov E, Smrkolj V. The Reliability and Medical Students’ Appreciation of Certainty-Based Marking. International Journal of Environmental Research and Public Health. 2022; 19(3):1706. https://doi.org/10.3390/ijerph19031706
Chicago/Turabian StyleSmrkolj, Špela, Enja Bančov, and Vladimir Smrkolj. 2022. "The Reliability and Medical Students’ Appreciation of Certainty-Based Marking" International Journal of Environmental Research and Public Health 19, no. 3: 1706. https://doi.org/10.3390/ijerph19031706