Journal of Medical Systems最新文献

筛选
英文 中文
Exploratory Implementation and Feasibility Report of CLASS (Clinical LLM Abstraction & Structuring System), A Large Language Model Pipeline for Extracting Unstructured Data From Clinical Notes. 从临床笔记中提取非结构化数据的大型语言模型管道CLASS (Clinical LLM Abstraction & Structuring System)的探索性实现及可行性报告。
IF 8.8 3区 医学
Journal of Medical Systems Pub Date : 2026-09-17 DOI: 10.1007/s10916-026-02462-6
Frederick H Kuo, Jamie L Fierstein, Fiorella Gonzales, Brant Tudor, Ellis Crabtree, Mohamed A Rehman, Russell Jennings, Luis Ahumada
{"title":"Exploratory Implementation and Feasibility Report of CLASS (Clinical LLM Abstraction & Structuring System), A Large Language Model Pipeline for Extracting Unstructured Data From Clinical Notes.","authors":"Frederick H Kuo, Jamie L Fierstein, Fiorella Gonzales, Brant Tudor, Ellis Crabtree, Mohamed A Rehman, Russell Jennings, Luis Ahumada","doi":"10.1007/s10916-026-02462-6","DOIUrl":"10.1007/s10916-026-02462-6","url":null,"abstract":"<p><p>Electronic medical records contain rich clinical detail, but much of it resides in unstructured notes, making manual abstraction for research and quality improvement labor-intensive and difficult to scale. This report describes the development of the Clinical LLM Abstraction & Structuring System (CLASS), a Python-based modular pipeline that uses a large language model (LLM) to extract structured data from clinical notes within a secure institutional environment. CLASS combines subject matter expert-curated concept lists, a task-specific prompt suite, and a schema-constrained output format, and exports results to a dashboard for expert review and analysis. The pipeline classifies predefined concepts and flags potential variants or novel concepts for expert consideration. Its modular design allows configuration for different abstraction tasks and can process large note volumes within standard institutional infrastructure. CLASS was applied to a retrospective corpus of pediatric esophageal airway treatment surgery (EATS) operative notes at a single center. In an exploratory evaluation comparing CLASS outputs against surgeon adjudication on the 20 longest notes (3,960 note-procedure pairs), observed concordance was high (F1 0.9967). CLASS also proposed 28 candidate procedure variants or additions, 18 (64.3%) of which were judged clinically useful, though the surgeon identified 12 additional procedures that CLASS did not surface. These findings suggest that an LLM-based pipeline can feasibly extract complex, non-coded procedure information from unstructured clinical notes within this specific single-center, single-service context, and may assist clinicians and informatics teams in curating and expanding specialized concept lists, though broader validation is needed to assess generalizability to other tasks and settings.</p>","PeriodicalId":16338,"journal":{"name":"Journal of Medical Systems","volume":"50 1","pages":""},"PeriodicalIF":8.8,"publicationDate":"2026-09-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148990345","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
CPS-Net (Collaborating Physicians in Silico Network): A Decentralized Multi-Agent Transformer Framework for Specialty-Aware Disease Prediction. CPS-Net(协作医生在计算机网络):一个分散的多智能体转换器框架,用于专科意识疾病预测。
IF 8.8 3区 医学
Journal of Medical Systems Pub Date : 2026-09-15 DOI: 10.1007/s10916-026-02460-8
Mohammad Assadi Shalmani, Masoud Khani, Michael S Harris, Qiang Lu, Jake Luo
{"title":"CPS-Net (Collaborating Physicians in Silico Network): A Decentralized Multi-Agent Transformer Framework for Specialty-Aware Disease Prediction.","authors":"Mohammad Assadi Shalmani, Masoud Khani, Michael S Harris, Qiang Lu, Jake Luo","doi":"10.1007/s10916-026-02460-8","DOIUrl":"10.1007/s10916-026-02460-8","url":null,"abstract":"<p><p>Diagnosis in real clinical settings is rarely finalized after a single encounter. Patients with complex conditions move through a referral network from primary care to specialists because physicians cannot maintain expertise across all diseases. Yet most diagnostic prediction tools apply a single generalist model across thousands of medical codes, which dilutes representational capacity for rare conditions and produces outputs poorly aligned with how clinicians actually work and reason. We present CPS-Net, a decentralized agent swarm that mirrors medical specialization hierarchies. The system connects specialized transformers trained on focused patient cohorts to language model agents organized into primary care, specialty, and subspecialty tiers. Agents collaborate without central supervision, communicating through shared case memory to make referral decisions transparent and auditable. We trained 41 transformers on Electronic Health Records (EHR) from 75,000 patients across 10 specialties and 30 subspecialties at Froedtert Hospital. Evaluated on 3,000 test cases, the framework achieved top-1, top-3, and top-5 accuracies of 48.8%, 71.00%, and 77.76% with 94.17% clinical relevance, substantially outperforming a monolithic transformer with 19.20%, 38.60% and 47.20% in top-1, top-3 and top-5 accuracy trained on all patients and a single language model agent baseline which achieved 12.70%, 26.40%, and 37.23% accuracy. This work demonstrates that aligning artificial intelligence architecture with clinical workflows improves both prediction accuracy and interpretability for diagnostic support.</p>","PeriodicalId":16338,"journal":{"name":"Journal of Medical Systems","volume":"50 1","pages":""},"PeriodicalIF":8.8,"publicationDate":"2026-09-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13578076/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148977055","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
A Flexible-Horizon Clinical Decision Support Model for Dementia Early Detection and Risk Prediction in Secondary Care: A Responsible AI Approach. 二级护理中痴呆早期检测和风险预测的灵活视界临床决策支持模型:一种负责任的人工智能方法。
IF 8.8 3区 医学
Journal of Medical Systems Pub Date : 2026-09-15 DOI: 10.1007/s10916-026-02458-2
Adnane Ez-Zizi, Kalyan Seelam, Lisa Leggett, Bilal R Malik
{"title":"A Flexible-Horizon Clinical Decision Support Model for Dementia Early Detection and Risk Prediction in Secondary Care: A Responsible AI Approach.","authors":"Adnane Ez-Zizi, Kalyan Seelam, Lisa Leggett, Bilal R Malik","doi":"10.1007/s10916-026-02458-2","DOIUrl":"10.1007/s10916-026-02458-2","url":null,"abstract":"<p><p>Existing machine learning (ML) models for dementia detection typically use fixed prediction windows and often lack clinically informed variable selection and responsible AI principles, thereby limiting performance, generalisability and clinical trust. We aimed to develop a single clinically informed model capable of both diagnosing dementia and predicting its onset over flexible time horizons up to 10 years. We trained our ML model on the NACC Uniform Data Set using 20 inputs compatible with UK National Healthcare Service secondary-care workflows, including a HORIZON feature encoding months from baseline. We also integrated explainability and fairness analyses. Same-visit classification achieved sensitivity of 0.951 (95% confidence interval 0.947-0.954), specificity of 0.810 (0.805-0.814), a geometric mean (G-mean) of 0.877 (0.874-0.880) and an area under the receiver operating characteristic curve (AUC) of 0.959 (0.957-0.960). Across future horizons from 24 to 120 months, sensitivity/specificity ranged between 0.808/0.856 at 24 months and 0.833/0.706 at 120 months, while AUCs were 0.906 and 0.851. Compared with a Cox proportional hazards model, our flexible-horizon ML model achieved a higher G-mean at four of the five evaluated horizons and higher specificity from 48 months onwards, whereas Cox achieved greater sensitivity from 48 months onwards. Shapley additive explanations highlighted functional independence, prediction horizon and cognitive measures as influential. Exploratory subgroup analyses showed broadly similar balanced discrimination across sex, race and age groups, although sensitivity-specificity trade-offs varied. An interactive web application was also developed for demonstration and clinical feedback. These findings support the feasibility of flexible-horizon, clinically-informed and responsible ML-based decision support for dementia early detection; however, external validation, recalibration and prospective clinical evaluation are required before deployment.</p>","PeriodicalId":16338,"journal":{"name":"Journal of Medical Systems","volume":"50 1","pages":""},"PeriodicalIF":8.8,"publicationDate":"2026-09-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13578025/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148977050","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Clinical AI and Precision Medicine: Philosophical Questions About Labels, Disease, and Evidence. 临床人工智能和精准医学:关于标签、疾病和证据的哲学问题。
IF 8.8 3区 医学
Journal of Medical Systems Pub Date : 2026-09-14 DOI: 10.1007/s10916-026-02457-3
Antonis A Armoundas
{"title":"Clinical AI and Precision Medicine: Philosophical Questions About Labels, Disease, and Evidence.","authors":"Antonis A Armoundas","doi":"10.1007/s10916-026-02457-3","DOIUrl":"10.1007/s10916-026-02457-3","url":null,"abstract":"","PeriodicalId":16338,"journal":{"name":"Journal of Medical Systems","volume":"50 1","pages":""},"PeriodicalIF":8.8,"publicationDate":"2026-09-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13572320/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148963396","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Machine Learning-based Early Detection of Intraoperative Anaphylaxis Among Patients with Hypotension Using Real-World Physiological Time Series Data. 使用真实世界生理时间序列数据进行低血压患者术中过敏反应的机器学习早期检测。
IF 8.8 3区 医学
Journal of Medical Systems Pub Date : 2026-09-04 DOI: 10.1007/s10916-026-02452-8
Haoran Su, Liang You, Bailin Jiang, Zilong Wu, Guilan Kong, Yi Feng
{"title":"Machine Learning-based Early Detection of Intraoperative Anaphylaxis Among Patients with Hypotension Using Real-World Physiological Time Series Data.","authors":"Haoran Su, Liang You, Bailin Jiang, Zilong Wu, Guilan Kong, Yi Feng","doi":"10.1007/s10916-026-02452-8","DOIUrl":"10.1007/s10916-026-02452-8","url":null,"abstract":"<p><strong>Background: </strong>Intraoperative anaphylaxis remains a rare yet fatal condition that has been a challenge due to its unpredictability. The unique characteristics of physiological parameter changes that precede or occur during the early stages of anaphylaxis may enable timely identification. We aimed to develop machine learning methods for the early detection of intraoperative anaphylaxis among patients with hypotension using real-world time-series physiological data.</p><p><strong>Methods: </strong>Physiological data, extracted from the EMRs at a 10-second sampling frequency, of patients undergoing surgeries at Peking University People's Hospital (January 1, 2011 - January 1, 2023) were analyzed. Three datasets (Datasets +1Min, +2Min, and +3Min) were constructed, each spanning 10 min before to 1, 2, and 3 min after hypotension, consisting of positive groups (intraoperative anaphylactic patients with hypotension) and negative groups (intraoperative non-anaphylactic patients with hypotension). Random forests, extreme gradient boosting, and categorical boosting (CatBoost) were employed to construct detection models. The model with the best performance was identified with a five-fold cross-validation.</p><p><strong>Results: </strong>Datasets +1Min, +2Min, and +3Min contained 49, 48, and 44 positive samples and 980, 960, and 880 negative samples, respectively. CatBoost models performed best on both Datasets +3Min and +2Min, achieving on Dataset +3Min an area under the receiver operator characteristic curve (AUROC) of 0.851 (± 0.082), an area under the precision-recall curve (AUPRC) of 0.431 (± 0.094), a sensitivity of 0.861 (± 0.210), and a specificity of 0.783 (± 0.164); corresponding values on Dataset +2Min were 0.823 (± 0.145), 0.365 (± 0.097), 0.711 (± 0.183), and 0.940 (± 0.065).</p><p><strong>Conclusions: </strong>This study suggests the feasibility of early detection of intraoperative anaphylaxis among patients with hypotension using physiological time-series data. The models could alert clinicians to suspected anaphylaxis just 2-3 min after hypotension, enabling early detection of intraoperative anaphylaxis.</p>","PeriodicalId":16338,"journal":{"name":"Journal of Medical Systems","volume":"50 1","pages":""},"PeriodicalIF":8.8,"publicationDate":"2026-09-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148891454","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Clarifying Associations Between HIS Provider Characteristics and Hospital Digital Maturity: Provider-Level Inference, Self-Inclusion, and Market-Size Dependence. 澄清医疗卫生服务提供者特征与医院数字化成熟度之间的关系:提供者水平推断、自我包容和市场规模依赖。
IF 8.8 3区 医学
Journal of Medical Systems Pub Date : 2026-09-04 DOI: 10.1007/s10916-026-02456-4
Hao Lyu, Yaowen Hu, Shucai Fan
{"title":"Clarifying Associations Between HIS Provider Characteristics and Hospital Digital Maturity: Provider-Level Inference, Self-Inclusion, and Market-Size Dependence.","authors":"Hao Lyu, Yaowen Hu, Shucai Fan","doi":"10.1007/s10916-026-02456-4","DOIUrl":"10.1007/s10916-026-02456-4","url":null,"abstract":"","PeriodicalId":16338,"journal":{"name":"Journal of Medical Systems","volume":"50 1","pages":""},"PeriodicalIF":8.8,"publicationDate":"2026-09-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148891507","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
An AI-assisted Clinical Decision Support System for Green Classification of Cystocele on Dynamic Transperineal Ultrasound. 人工智能辅助下经会阴动态超声胆囊膨出绿色分类的临床决策支持系统。
IF 8.8 3区 医学
Journal of Medical Systems Pub Date : 2026-09-01 DOI: 10.1007/s10916-026-02453-7
Hongjie Zhu, Xin Geng, Huayuan Zhou, Wei Guo, Yin Dai, Hao Zhang, Meng Dong, Hong Li, Xinlu Wang
{"title":"An AI-assisted Clinical Decision Support System for Green Classification of Cystocele on Dynamic Transperineal Ultrasound.","authors":"Hongjie Zhu, Xin Geng, Huayuan Zhou, Wei Guo, Yin Dai, Hao Zhang, Meng Dong, Hong Li, Xinlu Wang","doi":"10.1007/s10916-026-02453-7","DOIUrl":"10.1007/s10916-026-02453-7","url":null,"abstract":"<p><p>Green classification of cystocele on dynamic transperineal ultrasound (TPUS) remains operator-dependent because it requires manual frame selection and landmark-based assessment of the Valsalva maneuver. We developed a workflow-oriented AI-assisted clinical decision support system for automated urethrovesical junction localization and dynamic Green classification and prospectively evaluated its standalone and reader-support performance. This diagnostic accuracy and reader study included 881 patients from a tertiary referral hospital, comprising a retrospective development cohort (n = 688) and an independent prospective test cohort (n = 193). A nested subset of 67 prospective patients was used for a reader study involving two junior and two intermediate radiologists under unaided and AI-assisted conditions. In the complete prospective test cohort, Green-AttGRU achieved a macro-averaged AUC of 0.939 (95% CI, 0.897-0.971) and an overall accuracy of 0.902 (95% CI, 0.860-0.943). In the reader study, overall accuracy increased from 0.761 to 0.821 without AI to 0.851-0.881 with AI, while macro-F1 increased from 0.660 to 0.777 to 0.820-0.860. Overall inter-reader agreement increased from a Fleiss' κ of 0.453 to 0.786, and pooled median interpretation time decreased from 26.7 s to 9.9 s. These findings support the preliminary feasibility of the system as a workflow-oriented decision-support tool for dynamic TPUS interpretation.</p>","PeriodicalId":16338,"journal":{"name":"Journal of Medical Systems","volume":"50 1","pages":""},"PeriodicalIF":8.8,"publicationDate":"2026-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148874484","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Evaluation of MPEG-4 AAC Audio Codec Using EEG and EMG Signals for Use with DICOM® Neurophysiology. 利用脑电图和肌电图信号评价与DICOM®神经生理学使用的MPEG-4 AAC音频编解码器
IF 8.8 3区 医学
Journal of Medical Systems Pub Date : 2026-08-26 DOI: 10.1007/s10916-026-02451-9
Mattia Galanti, Filippo Battaglia, Giovanni Gugliandolo, Petter M Omland, Kristian B Nilsen, Sarah Breevoort, M Noreen Herring, Jerome Kurent, I-Hweii A Chen, Tuan Vu, Kyle Wuthrich, Nicola Donato, Mulugeta Gebregziabher, Giuseppe Campobello, Brian C Dean, Jonathan J Halford
{"title":"Evaluation of MPEG-4 AAC Audio Codec Using EEG and EMG Signals for Use with DICOM® Neurophysiology.","authors":"Mattia Galanti, Filippo Battaglia, Giovanni Gugliandolo, Petter M Omland, Kristian B Nilsen, Sarah Breevoort, M Noreen Herring, Jerome Kurent, I-Hweii A Chen, Tuan Vu, Kyle Wuthrich, Nicola Donato, Mulugeta Gebregziabher, Giuseppe Campobello, Brian C Dean, Jonathan J Halford","doi":"10.1007/s10916-026-02451-9","DOIUrl":"10.1007/s10916-026-02451-9","url":null,"abstract":"<p><p>Modern neurophysiological recording technologies are producing large volumes of data. This study investigates the use of the Moving Picture Experts Group version 4 Advanced Audio Coding (MPEG-4 AAC) for compressing electroencephalography (EEG) and electromyography (EMG) signals. All EEG signals contained either seizures or interictal epileptiform discharges (IEDs), which are nonstationary patterns characterized by sharp transients. Differences between pairs of original single-channel signals and compressed/decompressed signals were explored using the Percentage Root Mean Square Difference (PRD), power spectral density (PSD) features, and clinical expert opinion of waveform morphology. PRD was used as the primary error measure and PSD features were explored across different frequency bands. The results showed substantial decline in signal quality based on human expert opinion for compressed then reconstructed EEG signals with PRD error greater than 15% and EMG signals with PRD greater than 1%. PSD analysis showed significant changes in the gamma and beta bands for both EEG and EMG signals. This study provides further evidence that standard audio codecs which use psychoacoustic models, such as MPEG-4 AAC, do not sufficiently support compression of neurophysiology waveforms.</p>","PeriodicalId":16338,"journal":{"name":"Journal of Medical Systems","volume":"50 1","pages":""},"PeriodicalIF":8.8,"publicationDate":"2026-08-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13518441/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148828950","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Clinical AI Generators and Reviewers must be Tested Together. 临床AI生成器和审查器必须一起测试。
IF 8.8 3区 医学
Journal of Medical Systems Pub Date : 2026-08-25 DOI: 10.1007/s10916-026-02454-6
Vera Sorin, Eyal Klang
{"title":"Clinical AI Generators and Reviewers must be Tested Together.","authors":"Vera Sorin, Eyal Klang","doi":"10.1007/s10916-026-02454-6","DOIUrl":"10.1007/s10916-026-02454-6","url":null,"abstract":"","PeriodicalId":16338,"journal":{"name":"Journal of Medical Systems","volume":"50 1","pages":""},"PeriodicalIF":8.8,"publicationDate":"2026-08-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148818766","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Beyond Accuracy: A Mixed-Methods Audit of Chain-of-Thought Failures in LLM-Based COVID-19 Vaccine Stance Detection. 超越准确性:基于llm的COVID-19疫苗姿态检测中思维链失败的混合方法审计
IF 8.8 3区 医学
Journal of Medical Systems Pub Date : 2026-08-08 DOI: 10.1007/s10916-026-02449-3
Andreas Praschk, Valentin Fischill-Neudeck, Thomas Caspari, Hans-Peter Wiesinger
{"title":"Beyond Accuracy: A Mixed-Methods Audit of Chain-of-Thought Failures in LLM-Based COVID-19 Vaccine Stance Detection.","authors":"Andreas Praschk, Valentin Fischill-Neudeck, Thomas Caspari, Hans-Peter Wiesinger","doi":"10.1007/s10916-026-02449-3","DOIUrl":"10.1007/s10916-026-02449-3","url":null,"abstract":"<p><p>This mixed-methods study assessed whether reasoning-enabled large language models (LLMs) can classify stances towards COVID-19 vaccination on X (formerly Twitter) and whether model-generated chain-of-thought (CoT) summaries contain reasoning failures relevant to transparent and auditable public health applications. Zero-shot stance classification by o4-mini and Gemini 2.5 Flash (Gemini) was evaluated on 3,060 rehydrated COVID-19 vaccination tweets against human-annotated labels (positive, negative, neutral). We reported accuracy and macro-F1, measured CoT availability, and qualitatively analysed dual-error cases (tweets misclassified by both models) using Mayring's content analysis guided by the FUTURE-AI framework. At each model's best-performing setting, both models reached macro-F1 around 0.8, with o4-mini outperforming Gemini (accuracy 0.819 vs. 0.799, McNemar p = 0.0015; Δmacro-F1 = 0.020, 95% CI 0.008-0.032). Under the reasoning-intensive settings, CoT availability differed: Gemini returned a reasoning summary for all tweets, whereas o4-mini did so for 64.7%. Among 1,981 tweets with CoTs from both models, 295 (14.9%) were dual-errors; in 88.8%, both models produced the same wrong label, suggesting shared failure modes. Qualitatively, both models showed the same errors: target confusion (policy vs. vaccine), literal readings of sarcasm, and label-rationale mismatches, recurring across models despite their markedly different CoT lengths. Reasoning LLMs can therefore classify stance accurately, but their readiness for transparent public health applications depends on whether a CoT is available at all and whether it is coherent with the label it accompanies (label-rationale coherence). CoT availability, label-rationale coherence, and safeguards against systematic reasoning failures offer candidate explainability-readiness metrics, alongside accuracy, for trustworthy digital epidemiology.</p>","PeriodicalId":16338,"journal":{"name":"Journal of Medical Systems","volume":"50 1","pages":""},"PeriodicalIF":8.8,"publicationDate":"2026-08-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13452790/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148697557","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
相关产品
×
本文献相关产品
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信
小红书