Evaluating the Performance of Reasoning Large Language Models on Japanese Radiology Board Examination Questions.

IF 3.8 2区医学 Q1 RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING

Academic Radiology Pub Date : 2025-05-17 DOI:10.1016/j.acra.2025.04.060

Takeshi Nakaura, Hiroto Takamure, Naoki Kobayashi, Kaori Shiraishi, Naofumi Yoshida, Yasunori Nagayama, Hiroyuki Uetani, Masafumi Kidoh, Yoshinori Funama, Toshinori Hirai

{"title":"Evaluating the Performance of Reasoning Large Language Models on Japanese Radiology Board Examination Questions.","authors":"Takeshi Nakaura, Hiroto Takamure, Naoki Kobayashi, Kaori Shiraishi, Naofumi Yoshida, Yasunori Nagayama, Hiroyuki Uetani, Masafumi Kidoh, Yoshinori Funama, Toshinori Hirai","doi":"10.1016/j.acra.2025.04.060","DOIUrl":null,"url":null,"abstract":"Rationale and objectives: This study evaluates the performance, cost, and processing time of OpenAI's reasoning large language models (LLMs) (o1-preview, o1-mini) and their base models (GPT-4o, GPT-4o-mini) on Japanese radiology board examination questions.Materials and methods: A total of 210 questions from the 2022-2023 official board examinations of the Japan Radiological Society were presented to each of the four LLMs. Performance was evaluated by calculating the percentage of correctly answered questions within six predefined radiology subspecialties. The total cost and processing time for each model were also recorded. The McNemar test was used to assess the statistical significance of differences in accuracy between paired model responses.Results: The o1-preview achieved the highest accuracy (85.7%), significantly outperforming GPT-4o (73.3%, P<.001). Similarly, o1-mini (69.5%) performed significantly better than GPT-4o-mini (46.7%, P<.001). Across all radiology subspecialties, o1-preview consistently ranked highest. However, reasoning models incurred substantially higher costs (o1-preview: $17.10, o1-mini: $2.58) compared to their base counterparts (GPT-4o: $0.496, GPT-4o-mini: $0.04), and their processing times were approximately 3.7 and 1.2 times longer, respectively.Conclusion: Reasoning LLMs demonstrated markedly superior performance in answering radiology board exam questions compared to their base models, albeit at a substantially higher cost and increased processing time.","PeriodicalId":50928,"journal":{"name":"Academic Radiology","volume":" ","pages":""},"PeriodicalIF":3.8000,"publicationDate":"2025-05-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Academic Radiology","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1016/j.acra.2025.04.060","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING","Score":null,"Total":0}

引用次数: 0

Abstract

Rationale and objectives: This study evaluates the performance, cost, and processing time of OpenAI's reasoning large language models (LLMs) (o1-preview, o1-mini) and their base models (GPT-4o, GPT-4o-mini) on Japanese radiology board examination questions.

Materials and methods: A total of 210 questions from the 2022-2023 official board examinations of the Japan Radiological Society were presented to each of the four LLMs. Performance was evaluated by calculating the percentage of correctly answered questions within six predefined radiology subspecialties. The total cost and processing time for each model were also recorded. The McNemar test was used to assess the statistical significance of differences in accuracy between paired model responses.

Results: The o1-preview achieved the highest accuracy (85.7%), significantly outperforming GPT-4o (73.3%, P<.001). Similarly, o1-mini (69.5%) performed significantly better than GPT-4o-mini (46.7%, P<.001). Across all radiology subspecialties, o1-preview consistently ranked highest. However, reasoning models incurred substantially higher costs (o1-preview: $17.10, o1-mini: $2.58) compared to their base counterparts (GPT-4o: $0.496, GPT-4o-mini: $0.04), and their processing times were approximately 3.7 and 1.2 times longer, respectively.

Conclusion: Reasoning LLMs demonstrated markedly superior performance in answering radiology board exam questions compared to their base models, albeit at a substantially higher cost and increased processing time.

查看原文本刊更多论文

评估推理大型语言模型在日本放射学委员会考题中的表现。

基本原理和目的：本研究评估OpenAI推理大型语言模型（llm）（01 -preview, 01 -mini）及其基础模型（gpt - 40, gpt - 40 -mini）在日本放射学委员会考试问题上的性能、成本和处理时间。材料和方法：向四位法学硕士分别提交了日本放射学会2022-2023年官方委员会考试中的210道题。通过计算六个预定义的放射学亚专业中正确回答问题的百分比来评估绩效。还记录了每种型号的总成本和处理时间。使用McNemar检验来评估配对模型反应之间准确性差异的统计学意义。结果：1-预览达到了最高的准确率（85.7%），显著优于gpt - 40（73.3%）。结论：与基本模型相比，推理llm在回答放射学委员会考试问题方面表现出明显的优势，尽管成本更高，处理时间也更长。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Academic Radiology 医学-核医学

CiteScore

7.60

自引率

10.40%

发文量

432

审稿时长

18 days

期刊介绍： Academic Radiology publishes original reports of clinical and laboratory investigations in diagnostic imaging, the diagnostic use of radioactive isotopes, computed tomography, positron emission tomography, magnetic resonance imaging, ultrasound, digital subtraction angiography, image-guided interventions and related techniques. It also includes brief technical reports describing original observations, techniques, and instrumental developments; state-of-the-art reports on clinical issues, new technology and other topics of current medical importance; meta-analyses; scientific studies and opinions on radiologic education; and letters to the Editor.