Impact of AI assistance on pharmacist and pharmacy trainee accuracy and automation bias during medication order verification: A randomized simulation trial.

Abstract

Purpose: We evaluated the impact of artificial intelligence (AI) large language model (LLM) assistance on pharmacist accuracy during cognitive medication order verification and quantified the associated risks of automation bias.

Methods: Participants completed 10 clinical scenario questions through a custom web application, with each question randomly assigned to either an unassisted (control) or AI-assisted (intervention) arm. The intervention provided binary recommendations ("verify" or "not verify") and explanations generated by OpenAI's GPT-4o LLM. The primary outcome was user accuracy compared to a validated ground-truth standard.

Results: The study had a total of 77 participants, including 29 students, 14 postgraduate year 1 residents, 11 postgraduate year 2 (PGY2) residents, and 23 pharmacists. AI assistance was associated with significantly higher overall accuracy (adjusted odds ratio, 1.58; 95% confidence interval, 1.16-2.15). Subgroup analysis showed the greatest benefit among students and the least benefit among PGY2 residents. Performance was highly dependent on AI correctness: correct AI recommendations increased user accuracy to 80.2%, while incorrect recommendations reduced accuracy to 26.1%. Participants demonstrated high concordance with AI suggestions, accepting 80.2% of correct recommendations and 73.9% of incorrect recommendations. Notably, when users disagreed with the AI, their accuracy dropped to 21.7%, indicating that participants were rarely successful when attempting to override the model.

Conclusion: AI assistance enhances the accuracy of decisions by pharmacists and trainees, particularly for less experienced clinicians. However, the observed susceptibility to automation bias underscores that AI should augment, not replace, pharmacist clinical judgment. Safe integration requires pharmacist-guided model output validation and training focused on trust to ensure AI serves as a supportive adjunct rather than a substitute for professional accountability.