Implementation of a peer review process with faculty development in a pre-clinical didactic doctor of pharmacy curriculum to improve exam item quality: Lessons learned

Background

Exam items can contain unintended flaws that affect validity. Employing faculty development with an exam review panel may help address flaws and improve item quality. We evaluated item quality and perceptions after implementing a peer review process with faculty development in a Doctor of Pharmacy (PharmD) program.

Methods

In this pre-post study, lectures and online modules were developed and shared with faculty to enhance knowledge of best practice for item writing. Course directors recruited peers to review and improve items on summative assessments throughout the didactic curriculum in a PharmD program between academic years 2022–2024. Item quality on summative assessments before and after implementation was evaluated with the National Board of Medical Examination Item-Writing Guide and published checklist. In 2024, item writers, assessors, and course directors were surveyed to evaluate the effectiveness and challenges with the peer review process.

Results

A total of 488 items were evaluated (238 and 250 items before and after the implementation, respectively). The percentage of items with a flaw or an issue was lower by 6 % after the implementation (88.2 % vs. 82.2 %). Twenty-seven out of 32 participants (84 %) completed the survey. Seventy-six percent participated in the training and 64 % recruited peers. Challenges included buy-in, quality reviews, lack of accountability, and time management issues.

Conclusions

When implementing peer review with faculty development across the didactic curriculum in a PharmD program, pharmacy schools may need to develop strategies for faculty buy-in, recruitment and training of quality item assessors and timeline.