Evaluating the Accuracy of Large Language Models in Pharmaceutical Calculations: A Comparison of ChatGPT and MathGPT

IF 3.1 4区教育学 Q1 EDUCATION, SCIENTIFIC DISCIPLINES

American Journal of Pharmaceutical Education Pub Date : 2026-01-01 Epub Date: 2025-11-27 DOI:10.1016/j.ajpe.2025.101910

Bernadette Cornelison, Christopher Edwards, Crystal Zhang, Janae Dean, Julia Halloran, Aaron Middleton, Brian Erstad

{"title":"Evaluating the Accuracy of Large Language Models in Pharmaceutical Calculations: A Comparison of ChatGPT and MathGPT","authors":"Bernadette Cornelison, Christopher Edwards, Crystal Zhang, Janae Dean, Julia Halloran, Aaron Middleton, Brian Erstad","doi":"10.1016/j.ajpe.2025.101910","DOIUrl":null,"url":null,"abstract":"<div><h3>Objective</h3><div>Generative Artificial Intelligence is increasingly integrated into education, with large language models such as ChatGPT and MathGPT providing students with quick access to answers for pharmaceutical calculation problems. However, concerns persist regarding their accuracy for critical skills like pharmaceutical calculations. This study evaluates the accuracy of ChatGPT 3.5 and MathGPT Unlimited in solving pharmacy calculation problems relevant to student pharmacist curricula.</div></div><div><h3>Methods</h3><div>Fifty pharmaceutical calculation questions covering 9 North American Pharmacist Licensure Examination-aligned topic categories were selected from the University of Arizona R. Ken Coit College of Pharmacy curriculum. Responses generated by ChatGPT 3.5 (free version) and MathGPT Unlimited (data collection concluded July 2024) were compared against an instructor-validated answer key.</div></div><div><h3>Results</h3><div>Both ChatGPT and MathGPT correctly answered 35 of the 50 questions, with no significant difference between them. Accuracy varied across topics, with lower performance in sodium chloride Equivalence and Solutions. Logistic regression revealed no significant association between accuracy and question type or topic. Common errors included improper unit conversions, misapplication of calculation methods (eg, creatinine clearance with adjusted body weight), and variations in precision.</div></div><div><h3>Conclusion</h3><div>While demonstrating some capability, the 70% accuracy of ChatGPT 3.5 and MathGPT Unlimited, coupled with fundamental calculation errors, indicates they are insufficiently accurate for sole use in pharmaceutical calculations. These findings highlight the imperative for pharmacy educators to emphasize foundational skills and teach critical evaluation of artificial intelligence-generated responses.</div></div>","PeriodicalId":55530,"journal":{"name":"American Journal of Pharmaceutical Education","volume":"90 1","pages":"Article 101910"},"PeriodicalIF":3.1000,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"American Journal of Pharmaceutical Education","FirstCategoryId":"95","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S000294592500556X","RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2025/11/27 0:00:00","PubModel":"Epub","JCR":"Q1","JCRName":"EDUCATION, SCIENTIFIC DISCIPLINES","Score":null,"Total":0}

引用次数: 0

Abstract

Objective

Generative Artificial Intelligence is increasingly integrated into education, with large language models such as ChatGPT and MathGPT providing students with quick access to answers for pharmaceutical calculation problems. However, concerns persist regarding their accuracy for critical skills like pharmaceutical calculations. This study evaluates the accuracy of ChatGPT 3.5 and MathGPT Unlimited in solving pharmacy calculation problems relevant to student pharmacist curricula.

Methods

Fifty pharmaceutical calculation questions covering 9 North American Pharmacist Licensure Examination-aligned topic categories were selected from the University of Arizona R. Ken Coit College of Pharmacy curriculum. Responses generated by ChatGPT 3.5 (free version) and MathGPT Unlimited (data collection concluded July 2024) were compared against an instructor-validated answer key.

Results

Both ChatGPT and MathGPT correctly answered 35 of the 50 questions, with no significant difference between them. Accuracy varied across topics, with lower performance in sodium chloride Equivalence and Solutions. Logistic regression revealed no significant association between accuracy and question type or topic. Common errors included improper unit conversions, misapplication of calculation methods (eg, creatinine clearance with adjusted body weight), and variations in precision.

Conclusion

While demonstrating some capability, the 70% accuracy of ChatGPT 3.5 and MathGPT Unlimited, coupled with fundamental calculation errors, indicates they are insufficiently accurate for sole use in pharmaceutical calculations. These findings highlight the imperative for pharmacy educators to emphasize foundational skills and teach critical evaluation of artificial intelligence-generated responses.

查看原文本刊更多论文

评估大型语言模型在药物计算中的准确性：ChatGPT和MathGPT的比较。

目的：生成式人工智能（GenAI）越来越多地融入到教育中，ChatGPT和MathGPT等大型语言模型（llm）为学生提供了快速获取药物计算问题答案的途径。然而，对于诸如药物计算等关键技能的准确性，人们仍然存在担忧。本研究评估ChatGPT 3.5和MathGPT Unlimited在解决与药学专业课程相关的药学计算问题时的准确性。方法：从亚利桑那大学肯科伊特药学院的课程中选取50道药物计算题，涵盖9个与naplex对齐的主题类别。ChatGPT 3.5（免费版）和MathGPT Unlimited（数据收集于2024年7月结束）生成的答案与教师验证的答案键进行比较。结果：ChatGPT和MathGPT均正确回答了50道题中的35道，两者之间无显著差异。不同主题的准确性不同，在NaCl等效性和溶液中表现较差。逻辑回归显示准确率与问题类型或主题之间无显著关联。常见的错误包括不适当的单位转换，计算方法的错误应用（例如，与AdjBW的肌酐间隙），以及精度的变化。结论：虽然ChatGPT 3.5和MathGPT Unlimited显示出一定的能力，但70%的准确率加上基本的计算错误，表明它们不足以准确地用于药物计算。这些发现突出了药学教育工作者强调基本技能和教授对人工智能生成反应的批判性评估的必要性。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

American Journal of Pharmaceutical Education 医学-学科教育

CiteScore

4.30

自引率

15.20%

发文量

114

期刊介绍： The Journal accepts unsolicited manuscripts that have not been published and are not under consideration for publication elsewhere. The Journal only considers material related to pharmaceutical education for publication. Authors must prepare manuscripts to conform to the Journal style (Author Instructions). All manuscripts are subject to peer review and approval by the editor prior to acceptance for publication. Reviewers are assigned by the editor with the advice of the editorial board as needed. Manuscripts are submitted and processed online (Submit a Manuscript) using Editorial Manager, an online manuscript tracking system that facilitates communication between the editorial office, editor, associate editors, reviewers, and authors. After a manuscript is accepted, it is scheduled for publication in an upcoming issue of the Journal. All manuscripts are formatted and copyedited, and returned to the author for review and approval of the changes. Approximately 2 weeks prior to publication, the author receives an electronic proof of the article for final review and approval. Authors are not assessed page charges for publication.