Certainty-Based Marking in Multiple-Choice Assessments in Physiology: A Web-Based Implementation Using an AI Assistant.

Abstract

Certainty-based marking (CBM) requires students to indicate their certainty levels alongside their answers. CBM has been shown to enhance self-assessment and metacognitive awareness. This study aimed to explore the implementation of CBM in multiple-choice assessments in physiology. The CBM assessment tool was developed with an artificial intelligence (AI) assistant, Claude 3.5, with prompts focused on functional rather than technical requirements. The assessment consisted of 15 multiple-choice questions (MCQs), which were administered as a pretest and posttest during a small group teaching session to first-year medical students. Following the assessment, students completed a survey to evaluate their perceptions regarding the format, knowledge-gap identification, and overall acceptability. Answers from 195 students were analyzed, and significant improvements were observed in performance measures and certainty indices from the pretest to the posttest. Most students (80.9%) found the certainty scale beneficial, and 78.3% changed their answers after reflecting on their certainty. CBM demonstrated metacognitive benefits, with 86.4 % of students better recognizing their knowledge gaps and 85.8 % feeling more aware of their learning progress. About 73 % of students preferred the CBM format and expressed greater engagement (82.8 %) than traditional MCQs. CBM implemented through a web-based platform functioned as an assessment tool and an instructional intervention that enhances students' metacognitive awareness and self-monitoring skills in physiology education. Our study focused on a single physiology topic and showed improvements in knowledge retention and certainty calibration. However, further longitudinal studies across multiple topics are needed to determine whether students maintain these self-assessment skills over time.