Improving MCQ quality using a self-evaluation checklist: A quasi-experimental study

Abstract

Objective

To improve the quality of multiple-choice questions created by our faculty using a standardized checklist as an intervention.

Method

A pre–post quasi-experimental design was employed involving the faculty of basic medical sciences without a control group, to assess the impact of a self-evaluation checklist as an intervention tool on the quality of MCQs. Faculty members were familiarized with the predesigned checklist in an interactive training session. After the intervention, only self-evaluated MCQs were included. MCQ quality was measured through item analysis before and after the intervention. The Chi-square test was used to assess statistical significance.

Results

Using a checklist did not significantly improve MCQs' reliability, with a shift from “low” to “acceptable” internal exam consistency. There was a mixed response to the checklist in terms of the distribution of the difficulty levels of MCQs, with a slight increase in “difficult items” for anatomy and oral biology, a moderate decrease in “easy items” and a slight rise in “good/acceptable items” for physiology, and little to no change for biochemistry. Similarly, the results indicated that using a checklist decreased the discriminative ability of exam items in most subjects. However, a statistically significant improvement was observed in MCQs, with a proportional increase in the number of MCQs featuring “well-functioning distractors,” with increases of 26.43%, 12.5%, and 46.66% in the distractor efficiency for anatomy, biochemistry, and oral biology, respectively. Moreover, using a checklist improved the academic performance of students, particularly by reducing the number of students who performed “poorly” across most subjects, and significantly increasing the “good” ratings for anatomy, biochemistry, and oral biology, while more students achieved “satisfactory” in physiology.

Conclusion

Using a checklist for MCQ construction improved the quality of MCQs with well-functioning distractors, internal consistency of MCQs in exams, but had less impact on other item analysis parameters.