Benchmarking Vision Capabilities of Large Language Models in Surgical Examination Questions

IF 2.6 3区医学 Q1 EDUCATION, SCIENTIFIC DISCIPLINES

Journal of Surgical Education Pub Date : 2025-02-09 DOI:10.1016/j.jsurg.2025.103442

Jean-Paul Bereuter MD , Mark Enrik Geissler MS , Anna Klimova PhD , Robert-Patrick Steiner MD , Kevin Pfeiffer , Fiona R. Kolbinger MD , Isabella C. Wiest MD , Hannah Sophie Muti MD , Jakob Nikolas Kather MD

{"title":"Benchmarking Vision Capabilities of Large Language Models in Surgical Examination Questions","authors":"Jean-Paul Bereuter MD , Mark Enrik Geissler MS , Anna Klimova PhD , Robert-Patrick Steiner MD , Kevin Pfeiffer , Fiona R. Kolbinger MD , Isabella C. Wiest MD , Hannah Sophie Muti MD , Jakob Nikolas Kather MD","doi":"10.1016/j.jsurg.2025.103442","DOIUrl":null,"url":null,"abstract":"<div><h3>Objective</h3><div>Recent studies investigated the potential of large language models (LLMs) for clinical decision making and answering exam questions based on text input. Recent developments of LLMs have extended these models with vision capabilities. These image processing LLMs are called vision-language models (VLMs). However, there is limited investigation on the applicability of VLMs and their capabilities of answering exam questions with image content. Therefore, the aim of this study was to examine the performance of publicly accessible LLMs in 2 different surgical question sets consisting of text and image questions.</div></div><div><h3>Design</h3><div>Original text and image exam questions from 2 different surgical question subsets from the German Medical Licensing Examination (GMLE) and United States Medical Licensing Examination (USMLE) were collected and answered by publicly available LLMs (GPT-4, Claude-3 Sonnet, Gemini-1.5). LLM outputs were benchmarked for their accuracy in answering text and image questions. Additionally, the LLMs’ performance was compared to students’ performance based on their average historical performance (AHP) in these exams. Moreover, variations of LLM performance were analyzed in relation to question difficulty and respective image type.</div></div><div><h3>Results</h3><div>Overall, all LLMs achieved scores equivalent to passing grades (≥60%) on surgical text questions across both datasets. On image-based questions, only GPT-4 exceeded the score required to pass, significantly outperforming Claude-3 and Gemini-1.5 (GPT: 78% vs. Claude-3: 58% vs. Gemini-1.5: 57.3%; p < 0.001). Additionally, GPT-4 outperformed students on both text (GPT: 83.7% vs. AHP students: 67.8%; p < 0.001) and image questions (GPT: 78% vs. AHP students: 67.4%; p < 0.001).</div></div><div><h3>Conclusion</h3><div>GPT-4 demonstrated substantial capabilities in answering surgical text and image exam questions. Therefore, it holds considerable potential for the use in surgical decision making and education of students and trainee surgeons.</div></div>","PeriodicalId":50033,"journal":{"name":"Journal of Surgical Education","volume":"82 4","pages":"Article 103442"},"PeriodicalIF":2.6000,"publicationDate":"2025-02-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Surgical Education","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1931720425000236","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION, SCIENTIFIC DISCIPLINES","Score":null,"Total":0}

引用次数: 0

Abstract

Objective

Recent studies investigated the potential of large language models (LLMs) for clinical decision making and answering exam questions based on text input. Recent developments of LLMs have extended these models with vision capabilities. These image processing LLMs are called vision-language models (VLMs). However, there is limited investigation on the applicability of VLMs and their capabilities of answering exam questions with image content. Therefore, the aim of this study was to examine the performance of publicly accessible LLMs in 2 different surgical question sets consisting of text and image questions.

Design

Original text and image exam questions from 2 different surgical question subsets from the German Medical Licensing Examination (GMLE) and United States Medical Licensing Examination (USMLE) were collected and answered by publicly available LLMs (GPT-4, Claude-3 Sonnet, Gemini-1.5). LLM outputs were benchmarked for their accuracy in answering text and image questions. Additionally, the LLMs’ performance was compared to students’ performance based on their average historical performance (AHP) in these exams. Moreover, variations of LLM performance were analyzed in relation to question difficulty and respective image type.

Results

Overall, all LLMs achieved scores equivalent to passing grades (≥60%) on surgical text questions across both datasets. On image-based questions, only GPT-4 exceeded the score required to pass, significantly outperforming Claude-3 and Gemini-1.5 (GPT: 78% vs. Claude-3: 58% vs. Gemini-1.5: 57.3%; p < 0.001). Additionally, GPT-4 outperformed students on both text (GPT: 83.7% vs. AHP students: 67.8%; p < 0.001) and image questions (GPT: 78% vs. AHP students: 67.4%; p < 0.001).

Conclusion

GPT-4 demonstrated substantial capabilities in answering surgical text and image exam questions. Therefore, it holds considerable potential for the use in surgical decision making and education of students and trainee surgeons.

查看原文本刊更多论文

大型语言模型在外科检查问题中的视觉能力基准测试

目的最近的研究探讨了基于文本输入的大型语言模型（LLMs）在临床决策和回答考试问题方面的潜力。llm的最新发展已经用视觉功能扩展了这些模型。这些图像处理llm被称为视觉语言模型（vlm）。然而，关于vlm的适用性及其回答带有图像内容的考试问题的能力的研究有限。因此，本研究的目的是检验公开访问的法学硕士在由文本和图像问题组成的2个不同的外科问题集中的表现。设计从德国医学许可考试（GMLE）和美国医学许可考试（USMLE）的两个不同的外科问题子集中收集原始文本和图像试题，并由公开提供的法学硕士（GPT-4, Claude-3 Sonnet, Gemini-1.5）回答。法学硕士输出的基准是他们在回答文本和图像问题时的准确性。此外，根据学生在这些考试中的平均历史表现（AHP），将法学硕士的表现与学生的表现进行比较。此外，我们还分析了LLM表现与问题难度和各自图像类型的关系。结果总体而言，所有llm在两个数据集的外科文本问题上取得了相当于及格分数（≥60%）的分数。在基于图像的问题上，只有GPT-4超过了要求通过的分数，显著优于Claude-3和Gemini-1.5 (GPT: 78% vs. Claude-3: 58% vs. Gemini-1.5: 57.3%；p & lt;0.001)。此外，GPT-4学生在两个文本上的表现都优于学生(GPT: 83.7% vs. AHP学生：67.8%；p & lt;0.001)和形象问题(GPT: 78% vs. AHP学生：67.4%；p & lt;0.001)。结论pt -4在回答外科文本和图像检查问题方面表现出了很强的能力。因此，它在外科决策和学生和实习外科医生的教育中具有相当大的潜力。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of Surgical Education EDUCATION, SCIENTIFIC DISCIPLINES-SURGERY

CiteScore

5.60

自引率

10.30%

发文量

261

审稿时长

48 days

期刊介绍： The Journal of Surgical Education (JSE) is dedicated to advancing the field of surgical education through original research. The journal publishes research articles in all surgical disciplines on topics relative to the education of surgical students, residents, and fellows, as well as practicing surgeons. Our readers look to JSE for timely, innovative research findings from the international surgical education community. As the official journal of the Association of Program Directors in Surgery (APDS), JSE publishes the proceedings of the annual APDS meeting held during Surgery Education Week.