Large language models in spirometry interpretation: impact of guideline-based prompting.

IF 2.1 4区 医学 Q2 MEDICINE, GENERAL & INTERNAL
Croatian Medical Journal Pub Date : 2026-10-31
Muazzez Elçin Özkan, Damla Serçe Unat, Ömer Selim Unat, Mutlu Onur Güçsav, Mukaddes Hande Özgen, Dilara Özdemir, Erdal İlhan, Orhan Meral, Onur Akçay
{"title":"Large language models in spirometry interpretation: impact of guideline-based prompting.","authors":"Muazzez Elçin Özkan, Damla Serçe Unat, Ömer Selim Unat, Mutlu Onur Güçsav, Mukaddes Hande Özgen, Dilara Özdemir, Erdal İlhan, Orhan Meral, Onur Akçay","doi":"","DOIUrl":null,"url":null,"abstract":"<p><strong>Aim: </strong>To compare the performance of ChatGPT and Gemini with pulmonologist assessment in spirometry interpretation based on European Respiratory Society and American Thoracic Society (ERS/ATS) standards and to evaluate the effect of structured guideline-based prompting on diagnostic accuracy.</p><p><strong>Methods: </strong>This cross-sectional diagnostic study involved 500 anonymized spirometry reports. Quality control (QC) assessment and ventilatory pattern classification were based on pulmonologist consensus as the reference standard. The same cases were evaluated by ChatGPT and Gemini before and after structured guideline-based prompting derived from ERS/ATS criteria. Analyses were restricted to 362 tests deemed interpretable by all groups. Agreement was assessed with McNemar and McNemar-Bowker tests, and diagnostic performance was evaluated with class-specific and overall multiclass measures.</p><p><strong>Results: </strong>Both models correctly extracted all numerical spirometric parameters but overestimated test interpretability compared with specialists (all P<0.001). Among the 362 tests interpretable by all groups, ChatGPT's overall multiclass accuracy increased from 72.1% to 74.0% after prompting (κ=0.582 to 0.608), whereas Gemini's accuracy remained 69.9% (κ=0.552 in both phases). ChatGPT's sensitivity for obstructive patterns increased from 60.0% to 63.6%, whereas Gemini's decreased from 32.7% to 19.1%; sensitivity for mixed patterns remained low. Errors predominantly involved misclassification of obstructive patterns as restrictive or mixed.</p><p><strong>Conclusion: </strong>Guideline-based prompting produced model-specific and class-specific effects rather than a uniform improvement in diagnostic performance. Neither model reliably differentiated all ventilatory patterns. These findings support specialist-supervised, human-in-the-loop use and warrant standardized validation before clinical or medicolegal implementation.</p>","PeriodicalId":10796,"journal":{"name":"Croatian Medical Journal","volume":"67 5","pages":"338-346"},"PeriodicalIF":2.1000,"publicationDate":"2026-10-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Croatian Medical Journal","FirstCategoryId":"3","ListUrlMain":"","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"MEDICINE, GENERAL & INTERNAL","Score":null,"Total":0}
引用次数: 0

Abstract

Aim: To compare the performance of ChatGPT and Gemini with pulmonologist assessment in spirometry interpretation based on European Respiratory Society and American Thoracic Society (ERS/ATS) standards and to evaluate the effect of structured guideline-based prompting on diagnostic accuracy.

Methods: This cross-sectional diagnostic study involved 500 anonymized spirometry reports. Quality control (QC) assessment and ventilatory pattern classification were based on pulmonologist consensus as the reference standard. The same cases were evaluated by ChatGPT and Gemini before and after structured guideline-based prompting derived from ERS/ATS criteria. Analyses were restricted to 362 tests deemed interpretable by all groups. Agreement was assessed with McNemar and McNemar-Bowker tests, and diagnostic performance was evaluated with class-specific and overall multiclass measures.

Results: Both models correctly extracted all numerical spirometric parameters but overestimated test interpretability compared with specialists (all P<0.001). Among the 362 tests interpretable by all groups, ChatGPT's overall multiclass accuracy increased from 72.1% to 74.0% after prompting (κ=0.582 to 0.608), whereas Gemini's accuracy remained 69.9% (κ=0.552 in both phases). ChatGPT's sensitivity for obstructive patterns increased from 60.0% to 63.6%, whereas Gemini's decreased from 32.7% to 19.1%; sensitivity for mixed patterns remained low. Errors predominantly involved misclassification of obstructive patterns as restrictive or mixed.

Conclusion: Guideline-based prompting produced model-specific and class-specific effects rather than a uniform improvement in diagnostic performance. Neither model reliably differentiated all ventilatory patterns. These findings support specialist-supervised, human-in-the-loop use and warrant standardized validation before clinical or medicolegal implementation.

肺活量测定解释中的大型语言模型:基于指南提示的影响。
目的:比较ChatGPT和Gemini在基于欧洲呼吸学会(European Respiratory Society)和美国胸科学会(ERS/ATS)标准肺科医生评估肺活学解释中的表现,并评估基于指南的结构化提示对诊断准确性的影响。方法:这项横断面诊断研究包括500份匿名肺活量测定报告。质量控制(QC)评估和通气模式分类以肺科医师共识为参考标准。在基于ERS/ATS标准的结构化指南提示前后,使用ChatGPT和Gemini对相同的病例进行评估。分析仅限于362项被所有组认为可解释的测试。采用McNemar和McNemar- bowker试验评估一致性,并采用特定类别和总体多类别措施评估诊断性能。结果:两种模型都正确提取了所有数值肺活量测量参数,但与专家相比,高估了测试的可解释性(所有结论:基于指南的提示产生了特定模型和特定类别的效果,而不是诊断性能的统一改善。两个模型都不能可靠地区分所有的通气模式。这些发现支持专家监督、人在循环的使用,并保证在临床或医学法律实施之前进行标准化验证。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
Croatian Medical Journal
Croatian Medical Journal 医学-医学:内科
CiteScore
3.00
自引率
5.30%
发文量
105
审稿时长
6-12 weeks
期刊介绍: Croatian Medical Journal (CMJ) is an international peer reviewed journal open to scientists from all fields of biomedicine and health related research. Although CMJ welcomes all contributions that increase and expand on medical knowledge, the two areas are of the special interest: topics globally relevant for biomedicine and health and medicine in developing and emerging countries.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信
小红书