Using Machine Learning to Assess Factors Associated with North American Pharmacist Licensure Examination Performance.

Abstract

Objective: Pharmacy graduates' declining performance on the North American Pharmacist Licensure Examination (NAPLEX) remains concerning, but specific student factors related to success remain unclear. Machine learning (ML) algorithms may offer improved methods to identify potentially at-risk students before they take the examination.

Methods: Individuals graduating from the University of Kentucky College of Pharmacy in 2024 who passed (n=93) or failed (n=30) the NAPLEX on their first attempt were included. Over 20 characteristics related to demographics (e.g., age, sex, residence), undergraduate work (e.g., university, degree obtained, grade point average (GPA)), performance in the Doctor of Pharmacy program (e.g., GPA, elective courses taken, etc.), and engagement with NAPLEX preparatory software (RxPrep) were assessed for each student. CLASSify, a web-based platform for analysis of tabular data, was used to assess each of 8ML algorithms' ability to accurately predict whether a given student passed or failed. Area under the receiver operating curve (AUC-ROC) was primarily used to assess model accuracy. Average absolute SHapley Additive exPlanation (SHAP) value ranks were used to assess feature importance across models.

Results: Four of eight ML algorithms outperformed logistic regression (AUC-ROC 0.860), with the highest AUC-ROC in the random forest model (0.930). Across high performing models, the most important features were score on the college's high stakes progression examination (MileMarker 1), engagement with RxPrep, and traditional measures of academic performance.

Conclusion: ML algorithms accurately classified students' NAPLEX first-time performance in this cohort and could offer notable improvements to existing strategies colleges use to identify potentially at-risk students.