Tim Wilhelmi, Vanessa Bartsch, Marius Platt, Asarnusch Rashid, Johannes Hornig, Martin Krusche, Axel J Hueber, Daniel Fink, Alexander Pfeil, Gabriel Dischereit, Marie-Therese Holzer, Ulrich Drott, Peer Aries, Max Müller, Peter Böhm, Harriet Morf, Hannah Labinsky, Felix Mühlensiepen, Diego Benavent, Sebastian Kuhn, Philipp Klemm, Johannes Knitza
{"title":"Development and Nationwide Multicentre Evaluation of Guideline-Grounded Large Language Model Chatbots to Support Patient Self-Management and Education in Rheumatology.","authors":"Tim Wilhelmi, Vanessa Bartsch, Marius Platt, Asarnusch Rashid, Johannes Hornig, Martin Krusche, Axel J Hueber, Daniel Fink, Alexander Pfeil, Gabriel Dischereit, Marie-Therese Holzer, Ulrich Drott, Peer Aries, Max Müller, Peter Böhm, Harriet Morf, Hannah Labinsky, Felix Mühlensiepen, Diego Benavent, Sebastian Kuhn, Philipp Klemm, Johannes Knitza","doi":"10.1007/s10916-026-02470-6","DOIUrl":"https://doi.org/10.1007/s10916-026-02470-6","url":null,"abstract":"<p><p>Patients with rheumatic diseases have persistent information needs that are not fully addressed in routine care. We developed and evaluated guideline-grounded, large language model (LLM) chatbots to support patient self-management and education in rheumatology.Ten disease-specific chatbots based on German guidelines were co-developed and deployed through 13 rheumatology centres and six patient organisations. Chatbot users rated responses and completed a questionnaire. User questions, feedback, and response characteristics were analysed using category-based coding and a six-dimensional LLM-as-a-judge assessment, with LLM-based ratings compared with rheumatologist ratings in random subsets.Between September 2025 and January 2026, 6291 questions were recorded. Thirteen question categories were identified, most commonly disease-specific questions (50.2%), medication and monitoring (37.3%) and diagnostics (28.2%). The chatbots were unable to answer in 263 interactions (4.2%). Of 2671 responses rated by users, 2481 (92.9%) received a positive rating. Insufficient detail was the most common reason for negative ratings (125/190, 65.8%). Among 602 questionnaire respondents, 84.6% reported that the chatbot was easy to use, 84.1% that answers were easy to understand, and 80.2% that it was a useful addition to patient education. In the LLM-based evaluation, 95.3% of answers were rated as completely safe and 79.1% as completely correct. Guideline adherence was assessed separately, with 45.0% rated as fully adherent; agreement with physician assessment was weak.Guideline-grounded chatbots received predominantly positive user feedback in real-world use, while LLM-based evaluation suggested that most responses were safe and correct. User questions and feedback may help guide iterative improvements to source content and patient education materials. Further studies are needed to evaluate educational effectiveness and independently validate response quality and clinical safety.</p>","PeriodicalId":16338,"journal":{"name":"Journal of Medical Systems","volume":"50 1","pages":""},"PeriodicalIF":8.8,"publicationDate":"2026-10-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13635029/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149110924","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Špela Metličar, Kasper G Lauridsen, Gregor Štiglic, Nino Fijačko
{"title":"Association of Chest Compression Depth, Rate, and Spring Resistance with \"Quality CardioPulmonary Resuscitation\" (QCPR) Scores in a Manikin Model.","authors":"Špela Metličar, Kasper G Lauridsen, Gregor Štiglic, Nino Fijačko","doi":"10.1007/s10916-026-02469-z","DOIUrl":"https://doi.org/10.1007/s10916-026-02469-z","url":null,"abstract":"<p><p>Training in high-quality chest compressions (CCs) is essential for effective resuscitation; however, it remains unclear how accurately real-time feedback systems, such as Laerdal Medical's (Stavanger, Norway) Quality CardioPulmonary Resuscitation (QCPR) scoring system, reflect guideline-compliant CC quality. We aimed to investigate how CC depth, rate, and spring resistance affect QCPR scores. We conducted an experimental study using a Resusci Anne QCPR manikin (Laerdal Medical, Stavanger, Norway) equipped with interchangeable 30-, 50-, and 60-kg chest springs and a Corpuls mechanical CC device (GS Elektromedizinische Geräte G. Stemple GmbH, Kaufering, Germany). For each spring resistance, continuous CCs were delivered in two-minute intervals at predefined combinations of CC depth (2.0-6.0 cm) and rate (80-120 compressions per minute). CC quality was quantified using the QCPR score (0%-100%). Across all spring resistances, QCPR scores rose modestly at 3.0-3.5 cm and increased further at 4.0-4.5 cm, where the greatest score variation across preset CC rates and spring resistances was observed. At ≥ 5.0 cm, QCPR scores were consistently high (57%-100%). Overall, CC depth was the primary determinant of the QCPR score, with the highest scores observed at ≥ 5.0 cm. Additionally, we observed QCPR score differences of up to 18% for identical preset CC depth and rate settings with different springs. High QCPR scores were often achieved outside guideline ranges, emphasizing the need to interpret QCPR scores alongside absolute CC parameters and use guideline-compliant metrics as the standard for training and research.</p>","PeriodicalId":16338,"journal":{"name":"Journal of Medical Systems","volume":"50 1","pages":""},"PeriodicalIF":8.8,"publicationDate":"2026-10-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13635019/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149110960","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Hyeonhoon Lee, Seonhye Choi, Duyeon Kim, Kyunglan Hong, Hyeonsik Kim, Chang Wook Jeong, Hyung-Chul Lee
{"title":"Development and Validation of Human-AI Collaborative Workflow in SNOMED CT Mapping of Bilingual Clinical Text.","authors":"Hyeonhoon Lee, Seonhye Choi, Duyeon Kim, Kyunglan Hong, Hyeonsik Kim, Chang Wook Jeong, Hyung-Chul Lee","doi":"10.1007/s10916-026-02465-3","DOIUrl":"https://doi.org/10.1007/s10916-026-02465-3","url":null,"abstract":"<p><p>Systematized Nomenclature of Medicine-Clinical Terminology (SNOMED CT) is the principal international standard for semantic interoperability of clinical information, but mapping free-text clinical narratives to SNOMED CT concepts remains labor-intensive. We developed a large language model agent system for mapping bilingual clinical text to SNOMED CT concepts and evaluated its effect on mapping accuracy and efficiency within a human-AI collaborative workflow. We designed a three-module agent system comprising translation, abbreviation expansion, and vector-based retrieval components, integrated with a pre-embedded SNOMED CT vector database. Three health information managers independently mapped bilingual clinical text segments using three approaches: human-only, Agent-only, and Agent-assisted human mapping. Performance was evaluated by using hit rate, precision, recall, and F1 score at k = 1 and 5, and R-precision. Mapping time was compared between human-only and human-AI collaborative approaches. A total of 2,261 de-identified clinical text segments across nine clinical categories were collected at a tertiary academic hospital in South Korea. The human-AI collaborative workflow, which expanded the set of valid SNOMED CT candidates presented at each mapping decision, raised pooled hit rate@1 from 0.837 to 0.868 (difference 0.031, 95% confidence interval [CI] 0.021 to 0.042; p < 0.001), raised R-precision from 0.632 to 0.674 (difference 0.042, 95% CI 0.034 to 0.051; p < 0.001), and reduced total mapping time by 53.9% (from 1.57 to 0.72 min per segment, including agent processing). By expanding the space of valid SNOMED CT candidates available to expert mappers, the human-AI collaborative approach improved SNOMED CT mapping accuracy while reducing time by about half. Its modular architecture, supporting periodic vector database updates without retraining, offers a sustainable and efficient solution for bilingual clinical terminology standardization.</p>","PeriodicalId":16338,"journal":{"name":"Journal of Medical Systems","volume":"50 1","pages":""},"PeriodicalIF":8.8,"publicationDate":"2026-10-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13633333/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149110953","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Xia Wu, Xiaoli Shen, Changgeng Mo, Shengyan Shao, Jie Wang, Shangqiguo Wang
{"title":"Audiologist-Guided Multimodal AI for Pure-Tone Audiometry and Tympanometry Interpretation and Reporting.","authors":"Xia Wu, Xiaoli Shen, Changgeng Mo, Shengyan Shao, Jie Wang, Shangqiguo Wang","doi":"10.1007/s10916-026-02463-5","DOIUrl":"https://doi.org/10.1007/s10916-026-02463-5","url":null,"abstract":"<p><p>Multimodal artificial intelligence (AI) could support audiology reporting, but an end-to-end score can obscure whether errors arise from image transcription, specialty-rule execution, or communication. We evaluated these as separate modules in 155 de-identified outpatient records representing 151 patients. In Module 1, schema-only and audiologist-skill prompts were compared in 50 development records; the locked skill was then tested with a Codex GPT5.5 agentic workflow in 101 independent patients. A post hoc robustness analysis tested matched schema-only and skill-assisted direct-API conditions with DeepSeek-V4-Flash-Vision-Exp in the same 101 patients. The skill increased hearing-loss-type agreement from 85.0% to 94.0% and acoustic-reflex agreement from 70.4% to 98.4% in development. Codex validation yielded 1809/1820 (99.4%) exact numeric thresholds, but only 9/15 (60.0%) no-response entries were correct. In the DeepSeek robustness analysis, the skill increased type agreement from 37.6% to 70.3%, while exact numeric-threshold agreement remained 72.1% and no-response agreement remained 0%. In Module 2, AI workflows received adjudicated structured values, not Module 1 outputs. After two audiologists refined reports using 60 records, 91 independent patients were evaluated. All 25 rule-derived fields were correct in 91/91 Codex and 87/91 DeepSeek cases; both audiologists rated all Chinese reports accurate or basically accurate. These were different operational workflows, not a matched foundation-model comparison. The findings support prospective evaluation of modular, auditable clinician assistance, not autonomous diagnosis, end-to-end performance, or patient usability.</p>","PeriodicalId":16338,"journal":{"name":"Journal of Medical Systems","volume":"50 1","pages":""},"PeriodicalIF":8.8,"publicationDate":"2026-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13630873/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149110929","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Preserving Nursing Observations in Ambient AI Documentation on Psychiatric Wards.","authors":"Yushan Wei, Lien-Chung Wei","doi":"10.1007/s10916-026-02467-1","DOIUrl":"10.1007/s10916-026-02467-1","url":null,"abstract":"","PeriodicalId":16338,"journal":{"name":"Journal of Medical Systems","volume":"50 1","pages":""},"PeriodicalIF":8.8,"publicationDate":"2026-09-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149111002","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Beyond Regulatory Approval: Lifecycle Assurance for Clinical Artificial Intelligence.","authors":"Sohaib Shujaat","doi":"10.1007/s10916-026-02466-2","DOIUrl":"10.1007/s10916-026-02466-2","url":null,"abstract":"","PeriodicalId":16338,"journal":{"name":"Journal of Medical Systems","volume":"50 1","pages":""},"PeriodicalIF":8.8,"publicationDate":"2026-09-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149075053","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Ahmet Ziya Birbilen, Izzet Turkalp Akbasli, Ozlem Teksam
{"title":"Forecast-Driven Dynamic Physician Staffing in a Pediatric Emergency Department: A Prospective Quasi-Experimental Pilot Study.","authors":"Ahmet Ziya Birbilen, Izzet Turkalp Akbasli, Ozlem Teksam","doi":"10.1007/s10916-026-02464-4","DOIUrl":"10.1007/s10916-026-02464-4","url":null,"abstract":"<p><p>Emergency department crowding is a persistent threat to acute care quality, yet predictive models for ED demand have rarely been translated into prospective operational staffing interventions. Here we report a prospective single-center quasi-experimental pilot study evaluating forecast-driven dynamic physician scheduling in the pediatric ED of Hacettepe University Ihsan Dogramaci Children's Hospital (December 2024 to May 2025). Using a deep learning demand forecasting model (TiDE-RIN) combined with linear programming, we determined daily physician counts (range 3 to 6) for evening shifts (16:00 to 24:00) during days 1 to 15 of each month; days 16 to end of month maintained the institution's standard fixed four-physician schedule. Among after-hours visits with valid disposition timestamps (n = 9,626), mean post-evaluation length of stay (PE-LOS) was 175.9 min in the intervention arm and 184.0 min in the control arm (unadjusted difference 8.2 min; propensity-score-matched 7.9 min, median 10.0; stabilised inverse-probability weighting 8.5 min). Because staffing was allocated by calendar day, inference was clustered on the 182 study days: the two-way fixed-effects arm contrast was a reduction of 8.9 min (95% CI - 22.5 to + 4.8; p = 0.20) and the fully covariate-adjusted contrast 11.0 min (95% CI - 24.4 to + 2.5; p = 0.11). All design-consistent specifications placed the reduction between 4 and 11 min, none excluded no effect under day-level clustering, and the pilot was not powered for a difference of this size. Diagnostic testing was modestly lower (any-test rate 0.51 versus 0.54; p = 0.009). Effects were concentrated among lower-acuity patients and during the early-evening demand peak, and the optimised schedule allocated physicians demand-responsively (mean 4.31 versus 4.00 per shift) with no signal of compromised short-term patient safety, although per-physician workload effects did not reach significance in this pilot. Spillover analysis confirmed no progressive improvement in the concurrent control period. In this prospective pilot, coupling deep learning demand forecasting with optimisation-based physician scheduling was associated with a consistent but modest reduction in PE-LOS whose confidence interval included no effect, supporting feasibility and motivating multi-centre evaluation, and directly addressing the translational gap between predictive model development and real-world clinical implementation.</p>","PeriodicalId":16338,"journal":{"name":"Journal of Medical Systems","volume":"50 1","pages":""},"PeriodicalIF":8.8,"publicationDate":"2026-09-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149061682","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"IMPACT: Integrated Model For Patient Acuity, Care, And Triage Efficiency In Emergency Department.","authors":"Hamidreza Rasouli Panah, Abtin Ijadi Maghsoodi, Samaneh Madanian, Jian Yu","doi":"10.1007/s10916-026-02420-2","DOIUrl":"10.1007/s10916-026-02420-2","url":null,"abstract":"<p><p>Adverse patient outcomes in Emergency Departments (ED) are shaped by a complex interplay of clinical, demographic, and operational factors, yet existing approaches often fail to account for these dimensions in an integrated and actionable manner. This study introduces the Integrated Model for Patient Acuity, Care, and Triage Efficiency (IMPACT), a comprehensive framework developed to identify key predictors of mortality and generate individual adverse outcome risk scores across multiple clinical timeframes using a retrospective analytical approach. A retrospective analysis was conducted using 593,673 de-identified ED presentations from a tertiary hospital in the Waikato region in New Zealand, between 2016 and 2024. The modelling strategy involved a two-step approach, starting with Generalised Linear Models (GLM) for inferential analysis, followed by multiple regularised logistic regressions to enhance predictive accuracy and mitigate multicollinearity. The results indicated that advanced age, greater clinical complexity, and longer ED stays were associated with increased adverse outcomes, while longer waiting times were inversely associated with adverse outcomes, a finding consistent with triage-induced prioritisation effects whereby the most acutely unwell patients are seen first and also face the highest mortality risk, an associational pattern rather than a causal relationship. Based on the final model, an adverse outcome scoring system was derived, translating patient-level data into individual probability scores. The IMPACT model demonstrated high discriminative performance, with AUROC values of 0.9532 for ED mortality, 0.9183 for 10-days mortality, and 0.8999 for 28-days mortality. By combining robust statistical inference with a clinically interpretable scoring system, IMPACT provides a scalable and practical tool for retrospective risk stratification, service evaluation, and equitable decision-making in emergency care settings.</p>","PeriodicalId":16338,"journal":{"name":"Journal of Medical Systems","volume":"50 1","pages":""},"PeriodicalIF":8.8,"publicationDate":"2026-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13597584/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149025141","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Loris Azoyan, Judith Leblanc, Aline Santin, Christelle Chantalat-Auger, Sylvain Le Jeune, Louis Affo, Jean-Benoît Arlet, Pablo Bartolucci, Olivier Steichen
{"title":"Classifying Sickle Cell Disease Subtypes from Clinical Reports: Algorithm Validation and ICD-10 Accuracy Assessment in Five French Hospitals.","authors":"Loris Azoyan, Judith Leblanc, Aline Santin, Christelle Chantalat-Auger, Sylvain Le Jeune, Louis Affo, Jean-Benoît Arlet, Pablo Bartolucci, Olivier Steichen","doi":"10.1007/s10916-026-02461-7","DOIUrl":"10.1007/s10916-026-02461-7","url":null,"abstract":"<p><strong>Introduction: </strong>Research on rare diseases using large hospital and medico-administrative databases is expanding, yet precise patient characterization remains challenging. Sickle cell disease (SCD), given its different subtypes and the presence of sickle cell trait (SCT), illustrates these difficulties.</p><p><strong>Methods: </strong>We developed and validated an algorithm using regular expressions to classify SCD subtypes and SCT from unstructured clinical reports. We included all documents issued between 2013 and 2025 of adult patients with at least one ICD-10 code for SCD in five French expert centers of the Greater Paris University Hospitals. Manual review of a stratified random sample of 1510 patients served as the reference standard. We also evaluated the accuracy of various ICD-10-based selection strategies commonly used in medico-administrative studies to confirm SCD status.</p><p><strong>Results: </strong>Among the 10 868 included patients, 561 924 clinical reports were processed in five minutes using the algorithm. No sickle-related term was found for 1259 patients. Of the remaining 9 609 patients, 1 802 (18.8%) had non-specific sickle mentions, 1 502 (15.6%) were classified as SCT, 6 234 (64.9%) as SCD, of whom 4 443 (71.3%) were SS, 1 431 (23.0%) SC, 263 (4.2%) Sβ⁺ and 97 (1.6%) Sβ⁰. Finally, 71 (0.7%) were unresolved. The overall positive predictive value (PPV) of the algorithm was 94.0% (95% CI 92.6-95.4), ranging from 91.4% to 100% across subtypes. PPV of ICD-10 code-only strategies ranged from 58.1% to 88.3%, depending on the number of codes required and assumptions on unclassified patients.</p><p><strong>Conclusion: </strong>This simple algorithm effectively classifies SCD subtypes and SCT from clinical reports. Despite the large multicenter sample, performance may reflect the specific subtype demographics, expertise, and documentation practices of the study setting. This work highlights the risk of misclassification when relying solely on ICD-10 coding and the importance of high-quality clinical documentation for retrospective research.</p>","PeriodicalId":16338,"journal":{"name":"Journal of Medical Systems","volume":"50 1","pages":""},"PeriodicalIF":8.8,"publicationDate":"2026-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149025159","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Development of a Framework for Evaluating Large Language Model Safety and Reliability: a Proof-of-Concept Evaluation.","authors":"Fangyan Liu, Zhi Liu, Xiaolu Fei, Jingyu He, Jin Xing, Jia Li, Piu Chan","doi":"10.1007/s10916-026-02459-1","DOIUrl":"10.1007/s10916-026-02459-1","url":null,"abstract":"<p><p>Large language models (LLMs) are entering clinical decision support faster than methodology can characterise their safety. Aggregate accuracy treats all errors as interchangeable and cannot support safe deployment under Software as a Medical Device (SaMD) and EU AI Act frameworks. To develop and demonstrate a framework for evaluating large language model safety and reliability using catastrophic-failure frequency and response reproducibility, incorporating a pre-specified error taxonomy, difficulty-stratified analysis, and two-layer response consistency. The framework was applied to 54 diagnostically challenging emergency cases from non-public institutional records. Six LLMs were accessed via application programming interfaces (APIs) (text-only, zero-shot, defaults; August-December 2025) and queried three times each, producing 972 physician-scored responses. Aggregate scores concealed heterogeneity: anchoring-bias failure (score ≤ 3) varied nearly four-fold (6.9%-26.4%) and rare-disease recognition six-fold (6.1%-37.9%); catastrophic (dangerous-recommendation, score ≤ 2) rates were two-tiered (1.5% for the safest two vs. 6.0% for the rest; p < 0.001), though within-tier ranks were inseparable (2-12 per model). Despite a middle-tier mean, GPT-5.1 was least reproducible (within-case SD 2.69) and fell within the higher catastrophic-failure tier; across models, 31 of 34 dangerous combinations were stochastic, not systematic-a failure mode hidden by aggregate metrics. Open-source DeepSeek R1 was statistically equivalent to Gemini 3 Pro within a 1.5-point margin by two one-sided tests (TOST: Δ - 0.09; 90% CI - 0.47 to 0.30; p < 0.001). In this single-centre proof-of-concept, aggregate accuracy was insufficient for safety characterisation: models with indistinguishable mean accuracy carried different catastrophic-failure tiers and reproducibility profiles. Error-type and response-consistency profiling may inform model selection, ensemble design, and conformity assessment.</p>","PeriodicalId":16338,"journal":{"name":"Journal of Medical Systems","volume":"50 1","pages":""},"PeriodicalIF":8.8,"publicationDate":"2026-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13601161/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149030766","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}