Large language models for generating script concordance test in obstetrics and gynecology: ChatGPT and Claude.

IF 3.3 2区教育学 Q1 EDUCATION, SCIENTIFIC DISCIPLINES

Medical Teacher Pub Date : 2025-04-30 DOI:10.1080/0142159X.2025.2497888

Zuhal Yapıcı Coşkun, Yavuz Selim Kıyak, Özlem Coşkun, Işıl İrem Budakoğlu, Özhan Özdemir

{"title":"Large language models for generating script concordance test in obstetrics and gynecology: ChatGPT and Claude.","authors":"Zuhal Yapıcı Coşkun, Yavuz Selim Kıyak, Özlem Coşkun, Işıl İrem Budakoğlu, Özhan Özdemir","doi":"10.1080/0142159X.2025.2497888","DOIUrl":null,"url":null,"abstract":"Objective: To evaluate the performance of large language models (ChatGPT-4o and Claude 3.5 Sonnet) to generate script concordance test (SCT) items for assessing clinical reasoning in obstetrics and gynecology.Methods: This cross-sectional study involved the generation of SCT items for five common diagnostic topics in obstetrics and gynecology in primary care settings. A total of 16 panelists evaluated the AI-generated SCT items against 11 predefined criteria. Descriptive statistics were used to compare the models' performance across criteria.Results: ChatGPT-4o had an overall agreement rate of 90.57% for SCT items meeting the quality criteria, while Claude 3.5 Sonnet achieved 91.48%. The criterion with the lowest scores was \"The scenario is of appropriate difficulty for medical students,\" with ChatGPT-4o rated at 71.25% and Claude 3.5 Sonnet at 76.25%.Conclusion: Large language models can generate SCT items that effectively assess clinical reasoning; however, further refinement is required to ensure the appropriate level of difficulty for medical students. These findings highlight the potential of AI to enhance the efficiency of SCT generation in obstetrics and gynecology within primary care settings.","PeriodicalId":18643,"journal":{"name":"Medical Teacher","volume":" ","pages":"1-5"},"PeriodicalIF":3.3000,"publicationDate":"2025-04-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Medical Teacher","FirstCategoryId":"95","ListUrlMain":"https://doi.org/10.1080/0142159X.2025.2497888","RegionNum":2,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION, SCIENTIFIC DISCIPLINES","Score":null,"Total":0}

引用次数: 0

Abstract

Objective: To evaluate the performance of large language models (ChatGPT-4o and Claude 3.5 Sonnet) to generate script concordance test (SCT) items for assessing clinical reasoning in obstetrics and gynecology.

Methods: This cross-sectional study involved the generation of SCT items for five common diagnostic topics in obstetrics and gynecology in primary care settings. A total of 16 panelists evaluated the AI-generated SCT items against 11 predefined criteria. Descriptive statistics were used to compare the models' performance across criteria.

Results: ChatGPT-4o had an overall agreement rate of 90.57% for SCT items meeting the quality criteria, while Claude 3.5 Sonnet achieved 91.48%. The criterion with the lowest scores was "The scenario is of appropriate difficulty for medical students," with ChatGPT-4o rated at 71.25% and Claude 3.5 Sonnet at 76.25%.

Conclusion: Large language models can generate SCT items that effectively assess clinical reasoning; however, further refinement is required to ensure the appropriate level of difficulty for medical students. These findings highlight the potential of AI to enhance the efficiency of SCT generation in obstetrics and gynecology within primary care settings.

查看原文本刊更多论文

用于生成产科和妇科脚本一致性测试的大型语言模型：ChatGPT和Claude。

目的：评价大型语言模型（chatgpt - 40和Claude 3.5 Sonnet）生成用于评估妇产科临床推理的文字一致性测试（SCT）项目的性能。方法：本横断面研究涉及生成SCT项目的五个常见诊断主题的妇产科在初级保健机构。共有16名小组成员根据11个预定义标准评估人工智能生成的SCT项目。使用描述性统计来比较不同标准下模型的性能。结果：chatgpt - 40对符合质量标准的SCT项目的总体符合率为90.57%，而Claude 3.5 Sonnet的符合率为91.48%。得分最低的标准是“对医科学生来说难度适中”，chatgpt - 40的得分为71.25%，克劳德3.5十四行诗的得分为76.25%。结论：大型语言模型可以生成有效评估临床推理的SCT项目；然而，需要进一步完善，以确保适当的难度水平医学生。这些发现强调了人工智能在初级保健机构中提高产科和妇科SCT生成效率的潜力。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Medical Teacher 医学-卫生保健

CiteScore

7.80

自引率

8.50%

发文量

396

审稿时长

3-6 weeks

期刊介绍： Medical Teacher provides accounts of new teaching methods, guidance on structuring courses and assessing achievement, and serves as a forum for communication between medical teachers and those involved in general education. In particular, the journal recognizes the problems teachers have in keeping up-to-date with the developments in educational methods that lead to more effective teaching and learning at a time when the content of the curriculum—from medical procedures to policy changes in health care provision—is also changing. The journal features reports of innovation and research in medical education, case studies, survey articles, practical guidelines, reviews of current literature and book reviews. All articles are peer reviewed.