Comparison of Peer, Near-Peer, and AI-Assisted Methods to Faculty Grading of SOAP Notes.

IF 3.5 4区教育学 Q1 EDUCATION, SCIENTIFIC DISCIPLINES

American Journal of Pharmaceutical Education Pub Date : 2025-10-04 DOI:10.1016/j.ajpe.2025.101877

John Pham, Krista L Donohoe, Dayanjan Wijesinghe, Benjamin Van Tassell, Sarah E Wheeler, Apryl N Peddi

{"title":"Comparison of Peer, Near-Peer, and AI-Assisted Methods to Faculty Grading of SOAP Notes.","authors":"John Pham, Krista L Donohoe, Dayanjan Wijesinghe, Benjamin Van Tassell, Sarah E Wheeler, Apryl N Peddi","doi":"10.1016/j.ajpe.2025.101877","DOIUrl":null,"url":null,"abstract":"Objective: The objective of this study was to determine which approach (peer, near-peer, ChatGPT Plus, or an in-house implementation of a GPT-4-based application referred to as GRADES) was the closest to faculty grading of SOAP notes.Methods: Second-year pharmacy students (n=83) completed a practice SOAP note. Five methods were used to grade the SOAP notes: faculty, peer, near-peer, ChatGPT Plus, and GRADES. Variability in rubric scores among grading methods was analyzed using Friedman one-way repeated measures ANOVA with post-hoc testing. Time was tracked for each grading method to evaluate efficiency.Results: The median scores and interquartile ranges for each grading method were as follows: faculty (65% [56% - 71%], peer (78% [62% - 88%]), near-peer (77% [72% - 86%]), ChatGPT Plus (87% [80% - 91%]), and GRADES (66% [57% - 73%]). Peer-, near-peer, and ChatGPT scores were statistically different from faculty scores. Scores from GRADES were not significantly different from faculty grading.Conclusion: The use of an in-house implementation of GPT-4 (GRADES), with faculty oversight, resulted in similar rubric scores to a faculty grader.","PeriodicalId":55530,"journal":{"name":"American Journal of Pharmaceutical Education","volume":" ","pages":"101877"},"PeriodicalIF":3.5000,"publicationDate":"2025-10-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"American Journal of Pharmaceutical Education","FirstCategoryId":"95","ListUrlMain":"https://doi.org/10.1016/j.ajpe.2025.101877","RegionNum":4,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION, SCIENTIFIC DISCIPLINES","Score":null,"Total":0}

引用次数: 0

Abstract

Objective: The objective of this study was to determine which approach (peer, near-peer, ChatGPT Plus, or an in-house implementation of a GPT-4-based application referred to as GRADES) was the closest to faculty grading of SOAP notes.

Methods: Second-year pharmacy students (n=83) completed a practice SOAP note. Five methods were used to grade the SOAP notes: faculty, peer, near-peer, ChatGPT Plus, and GRADES. Variability in rubric scores among grading methods was analyzed using Friedman one-way repeated measures ANOVA with post-hoc testing. Time was tracked for each grading method to evaluate efficiency.

Results: The median scores and interquartile ranges for each grading method were as follows: faculty (65% [56% - 71%], peer (78% [62% - 88%]), near-peer (77% [72% - 86%]), ChatGPT Plus (87% [80% - 91%]), and GRADES (66% [57% - 73%]). Peer-, near-peer, and ChatGPT scores were statistically different from faculty scores. Scores from GRADES were not significantly different from faculty grading.

Conclusion: The use of an in-house implementation of GPT-4 (GRADES), with faculty oversight, resulted in similar rubric scores to a faculty grader.

查看原文本刊更多论文

对等、近对等和人工智能辅助方法对教师SOAP笔记评分的比较。

目的：本研究的目的是确定哪种方法（同级、近同级、ChatGPT Plus，或者内部实现的基于gpt -4的应用程序，称为GRADES）最接近SOAP笔记的教师评分。方法：药学二年级学生（83名）完成一份SOAP实践笔记。我们使用了五种方法对SOAP笔记进行评分：教员、同行、近同行、ChatGPT Plus和评分。采用Friedman单向重复测量方差分析和事后检验分析了分级方法中分类分数的可变性。跟踪每一种分级方法的时间，以评估效率。结果：每种评分方法的中位数得分和四分位数范围如下：教师（65%[56% - 71%]），同伴（78%[62% - 88%]），近同伴（77% [72% - 86%]），ChatGPT Plus（87%[80% - 91%]）和等级（66%[57% - 73%]）。同伴、近同伴和ChatGPT分数与教师分数有统计学差异。年级评分与教师评分无显著差异。结论：在教师监督下，使用内部实施的GPT-4 (grade)，产生了与教师评分相近的标题分数。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

American Journal of Pharmaceutical Education 医学-学科教育

CiteScore

4.30

自引率

15.20%

发文量

114

期刊介绍： The Journal accepts unsolicited manuscripts that have not been published and are not under consideration for publication elsewhere. The Journal only considers material related to pharmaceutical education for publication. Authors must prepare manuscripts to conform to the Journal style (Author Instructions). All manuscripts are subject to peer review and approval by the editor prior to acceptance for publication. Reviewers are assigned by the editor with the advice of the editorial board as needed. Manuscripts are submitted and processed online (Submit a Manuscript) using Editorial Manager, an online manuscript tracking system that facilitates communication between the editorial office, editor, associate editors, reviewers, and authors. After a manuscript is accepted, it is scheduled for publication in an upcoming issue of the Journal. All manuscripts are formatted and copyedited, and returned to the author for review and approval of the changes. Approximately 2 weeks prior to publication, the author receives an electronic proof of the article for final review and approval. Authors are not assessed page charges for publication.