Muazzez Elçin Özkan, Damla Serçe Unat, Ömer Selim Unat, Mutlu Onur Güçsav, Mukaddes Hande Özgen, Dilara Özdemir, Erdal İlhan, Orhan Meral, Onur Akçay
{"title":"Large language models in spirometry interpretation: impact of guideline-based prompting.","authors":"Muazzez Elçin Özkan, Damla Serçe Unat, Ömer Selim Unat, Mutlu Onur Güçsav, Mukaddes Hande Özgen, Dilara Özdemir, Erdal İlhan, Orhan Meral, Onur Akçay","doi":"","DOIUrl":null,"url":null,"abstract":"<p><strong>Aim: </strong>To compare the performance of ChatGPT and Gemini with pulmonologist assessment in spirometry interpretation based on European Respiratory Society and American Thoracic Society (ERS/ATS) standards and to evaluate the effect of structured guideline-based prompting on diagnostic accuracy.</p><p><strong>Methods: </strong>This cross-sectional diagnostic study involved 500 anonymized spirometry reports. Quality control (QC) assessment and ventilatory pattern classification were based on pulmonologist consensus as the reference standard. The same cases were evaluated by ChatGPT and Gemini before and after structured guideline-based prompting derived from ERS/ATS criteria. Analyses were restricted to 362 tests deemed interpretable by all groups. Agreement was assessed with McNemar and McNemar-Bowker tests, and diagnostic performance was evaluated with class-specific and overall multiclass measures.</p><p><strong>Results: </strong>Both models correctly extracted all numerical spirometric parameters but overestimated test interpretability compared with specialists (all P<0.001). Among the 362 tests interpretable by all groups, ChatGPT's overall multiclass accuracy increased from 72.1% to 74.0% after prompting (κ=0.582 to 0.608), whereas Gemini's accuracy remained 69.9% (κ=0.552 in both phases). ChatGPT's sensitivity for obstructive patterns increased from 60.0% to 63.6%, whereas Gemini's decreased from 32.7% to 19.1%; sensitivity for mixed patterns remained low. Errors predominantly involved misclassification of obstructive patterns as restrictive or mixed.</p><p><strong>Conclusion: </strong>Guideline-based prompting produced model-specific and class-specific effects rather than a uniform improvement in diagnostic performance. Neither model reliably differentiated all ventilatory patterns. These findings support specialist-supervised, human-in-the-loop use and warrant standardized validation before clinical or medicolegal implementation.</p>","PeriodicalId":10796,"journal":{"name":"Croatian Medical Journal","volume":"67 5","pages":"338-346"},"PeriodicalIF":2.1000,"publicationDate":"2026-10-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Croatian Medical Journal","FirstCategoryId":"3","ListUrlMain":"","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"MEDICINE, GENERAL & INTERNAL","Score":null,"Total":0}
引用次数: 0
Abstract
Aim: To compare the performance of ChatGPT and Gemini with pulmonologist assessment in spirometry interpretation based on European Respiratory Society and American Thoracic Society (ERS/ATS) standards and to evaluate the effect of structured guideline-based prompting on diagnostic accuracy.
Methods: This cross-sectional diagnostic study involved 500 anonymized spirometry reports. Quality control (QC) assessment and ventilatory pattern classification were based on pulmonologist consensus as the reference standard. The same cases were evaluated by ChatGPT and Gemini before and after structured guideline-based prompting derived from ERS/ATS criteria. Analyses were restricted to 362 tests deemed interpretable by all groups. Agreement was assessed with McNemar and McNemar-Bowker tests, and diagnostic performance was evaluated with class-specific and overall multiclass measures.
Results: Both models correctly extracted all numerical spirometric parameters but overestimated test interpretability compared with specialists (all P<0.001). Among the 362 tests interpretable by all groups, ChatGPT's overall multiclass accuracy increased from 72.1% to 74.0% after prompting (κ=0.582 to 0.608), whereas Gemini's accuracy remained 69.9% (κ=0.552 in both phases). ChatGPT's sensitivity for obstructive patterns increased from 60.0% to 63.6%, whereas Gemini's decreased from 32.7% to 19.1%; sensitivity for mixed patterns remained low. Errors predominantly involved misclassification of obstructive patterns as restrictive or mixed.
Conclusion: Guideline-based prompting produced model-specific and class-specific effects rather than a uniform improvement in diagnostic performance. Neither model reliably differentiated all ventilatory patterns. These findings support specialist-supervised, human-in-the-loop use and warrant standardized validation before clinical or medicolegal implementation.
期刊介绍:
Croatian Medical Journal (CMJ) is an international peer reviewed journal open to scientists from all fields of biomedicine and health related research.
Although CMJ welcomes all contributions that increase and expand on medical knowledge, the two areas are of the special interest: topics globally relevant for biomedicine and health and medicine in developing and emerging countries.