Journal of the American Medical Informatics Association最新文献

{"title":"Assessing and improving data quality in health registries: a systematic review of dimensions, frameworks, and artificial intelligence-enabled approaches.","authors":"Seyyedeh Fatemeh Mousavi Baigi, Seyyed Mohammad Tabatabaei, Abbas Sheikhtaheri, Masoumeh Sarbaz, Khalil Kimiafar","doi":"10.1093/jamia/ocag149","DOIUrl":"https://doi.org/10.1093/jamia/ocag149","url":null,"abstract":"<p><strong>Objectives: </strong>To systematically synthesize how data quality (DQ) in disease and health outcomes registries has been conceptualized, assessed, and improved, emphasizing artificial intelligence (AI) and machine learning methods within registry-based DQ workflows.</p><p><strong>Materials and methods: </strong>Following PRISMA 2020 guidelines, a comprehensive search was conducted across PubMed, Embase, Scopus, Web of Science, IEEE Xplore, Google Scholar, and Google through October 26, 2025. Eligible studies evaluated DQ dimensions, frameworks, methods, tools, or algorithms in structured health registries using conventional or AI-based approaches. Methodological quality was appraised using the Mixed Methods Appraisal Tool. Findings were synthesized using qualitative thematic analysis.</p><p><strong>Results: </strong>A total of 119 studies were included. Completeness (111 studies, 93.3%) and accuracy (93 studies, 78.2%) were the most frequently assessed DQ dimensions, whereas representativeness and coverage were less frequently evaluated. Conventional rule-based, statistical, and audit approaches remained dominant. Artificial intelligence/machine learning methods were applied selectively for anomaly detection, missing-data tasks, record linkage, and automated data extraction. However, implementation remained constrained by heterogeneous data structures, limited interoperability, and insufficient reference standards. Reported limitations most commonly involved methodological and analytical constraints, incomplete data capture, and restricted source verification. Frameworks were predominantly assessment-oriented, with limited integration of lifecycle-based quality management strategies.</p><p><strong>Discussion: </strong>Despite increasing interest in AI-enabled DQ improvement, registry quality management remains retrospective, fragmented, and insufficiently integrated with lifecycle-oriented infrastructures.</p><p><strong>Conclusion: </strong>Advancing registry DQ requires a transition from retrospective validation toward proactive, lifecycle-integrated, and AI-enabled quality ecosystems. Registry-specific frameworks, interoperable infrastructures, and benchmarking strategies are essential to support scalable and trustworthy registry data systems.</p><p><strong>Prospero registration id: </strong>CRD420251176960 (https://www.crd.york.ac.uk/PROSPERO/view/CRD420251176960).</p>","PeriodicalId":50016,"journal":{"name":"Journal of the American Medical Informatics Association","volume":" ","pages":""},"PeriodicalIF":7.1,"publicationDate":"2026-09-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149082592","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"MADE: leveraging informative missingness and temporal evaluation for electronic health record-based deterioration prediction in general wards.","authors":"Yechan Mun, Jee Hwan Ahn, Chang Youl Lee, Byungeun Ahn, Jin Won Huh, Kyung Soo Chung","doi":"10.1093/jamia/ocag124","DOIUrl":"https://doi.org/10.1093/jamia/ocag124","url":null,"abstract":"<p><strong>Objective: </strong>Electronic health record (EHR) time series are irregularly sampled and frequently incomplete, with missingness potentially carrying clinically informative signals. We developed Masked-Attention-based timestamp-wise Data Embedding (MADE), a missingness-aware model for continuous prediction of deterioration in general wards, and evaluated it across heterogeneous hospitals.</p><p><strong>Materials and methods: </strong>Observed variables were tokenized into feature-specific embeddings and processed by a masked-attention Transformer, followed by bi-LSTM temporal aggregation for risk prediction. MADE was evaluated for major adverse event (MAE) and sepsis in 1 internal (SVH) and 2 external cohorts (AMC, HUMC) against conventional and deep learning baselines. Ablation studies compared no imputation with last observation carried forward (LOCF) and multiple imputations by chained equations (MICE). Discriminative performance was assessed using the area under the receiver operating characteristic curve (AUROC) and precision-recall curve (AUPRC). Clinical utility and reliability were assessed using decision curve analysis, Brier score, and expected calibration error (ECE).</p><p><strong>Results: </strong>MADE achieved strong discrimination for MAE (AUROC 0.966/0.902/0.825 for SVH/AMC/HUMC) and sepsis (AUROC 0.934/0.874/0.776). The no-imputation configuration generally outperformed LOCF and MICE, particularly for MAE and for sepsis AUPRC, supporting preservation of informative missingness. Decision curves showed favorable net benefit for MAE and competitive benefit for sepsis. Calibration indicated modest underestimation at higher risks, suggesting site-specific recalibration.</p><p><strong>Discussion: </strong>MADE mitigates limitations of fixed grid and imputation-dependent EHR modeling by jointly leveraging observed values and missingness patterns under heterogeneous sampling.</p><p><strong>Conclusion: </strong>MADE supports scalable observation-aligned risk monitoring, particularly for MAE, while sepsis prediction may require site-specific recalibration or feature-set adaptation under cross-institution domain shifts.</p>","PeriodicalId":50016,"journal":{"name":"Journal of the American Medical Informatics Association","volume":" ","pages":""},"PeriodicalIF":7.1,"publicationDate":"2026-09-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149082595","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"What do we assess when we assess mHealth app privacy and security? A rapid scoping review of mHealth app privacy instruments.","authors":"Fangziyun Tong, Stella Antoniou, Mark Larsen","doi":"10.1093/jamia/ocag154","DOIUrl":"https://doi.org/10.1093/jamia/ocag154","url":null,"abstract":"<p><strong>Objectives: </strong>While there are numerous mobile health (mHealth) apps on the market, previous research has highlighted their risks to privacy and security. Therefore, an instrument to assess the privacy and security of these apps may be helpful for ensuring consumers' data safety. We therefore conducted this rapid scoping review with the objective of surveying the landscape for instruments used to measure and quantify privacy and/or security of mHealth apps in the past 10 years.</p><p><strong>Materials and methods: </strong>Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 guidelines, we systematically searched the relevant literature in 4 electronic databases between May 2015 and May 2025. The characteristics of the papers and/or instruments were analyzed and reported descriptively. The instruments' content was analyzed using content analysis.</p><p><strong>Results: </strong>We identified 66 papers describing 42 distinct instruments. We found that the number of items and rating methods varied significantly across the instruments. Five topics were identified from content analysis: data practice, security measures, risk and incident management, regulatory compliance and users' rights, and privacy policy and terms and conditions.</p><p><strong>Discussion: </strong>While the nature of the instruments is diverse, it is currently unknown how many items, which topics, or how responses should be structured is optimal. Additionally, the majority of identified instruments cannot comprehensively measure the proposed 5 topics. Moreover, questions on security measures may not be easily rated by users without specific domain knowledge.</p><p><strong>Conclusion: </strong>This review can serve as a guide for future mHealth app privacy and security evaluation.</p>","PeriodicalId":50016,"journal":{"name":"Journal of the American Medical Informatics Association","volume":" ","pages":""},"PeriodicalIF":7.1,"publicationDate":"2026-09-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149082663","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Posthumous health data donation: an emerging framework for equitable healthcare AI.","authors":"Pallavi Gupta, Maxim Topaz, Lisiane Pruinelli, Anji Wall, Charlene E Ronquillo, Martin Michalowski, Jing Dong, Zhihong Zhang, Laura-Maria Peltonen","doi":"10.1093/jamia/ocag156","DOIUrl":"https://doi.org/10.1093/jamia/ocag156","url":null,"abstract":"<p><strong>Objective: </strong>To examine posthumous health data donation as a pathway toward more representative datasets used to develop healthcare artificial intelligence (AI).</p><p><strong>Perspective: </strong>Healthcare AI systems are increasingly deployed in clinical settings, yet the datasets underlying these systems remain narrow, fragmented, and systematically underrepresenting key patient populations. Current consent frameworks, designed for living patients, create structural barriers to assembling diverse training data. Death meaningfully shifts the ethical risk-benefit calculus, substantially reducing harms, employment discrimination, insurance denial, social stigma, that presuppose a living subject. This creates a unique opportunity for individuals to contribute their health data posthumously through informed, opt-in consent established during life.</p><p><strong>Conclusion: </strong>Posthumous health data donation warrants serious exploration by the informatics, policy, and governance communities as a complementary mechanism for reducing structural barriers to inclusion, one that does not guarantee representativeness but may address gaps that living-patient consent frameworks cannot reach.</p>","PeriodicalId":50016,"journal":{"name":"Journal of the American Medical Informatics Association","volume":" ","pages":""},"PeriodicalIF":7.1,"publicationDate":"2026-09-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149082553","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Correction to: The Potential Implications of Informatics for Value-Based Care.","authors":"","doi":"10.1093/jamia/ocag165","DOIUrl":"https://doi.org/10.1093/jamia/ocag165","url":null,"abstract":"","PeriodicalId":50016,"journal":{"name":"Journal of the American Medical Informatics Association","volume":" ","pages":""},"PeriodicalIF":7.1,"publicationDate":"2026-09-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149070499","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Transporting a pediatric surgical site infection prediction model to the electronic health record: external validation, harmonization, and recalibration.","authors":"Carrie T Chan, Mark J Pletcher, Karthik Balakrishnan, Yulin Hswen, Aaron Scheffler","doi":"10.1093/jamia/ocag162","DOIUrl":"https://doi.org/10.1093/jamia/ocag162","url":null,"abstract":"<p><strong>Objectives: </strong>To externally validate a pediatric surgical site infection (SSI) prediction model, assess institutional and temporal transportability, and evaluate recalibration strategies to support adaptation of registry-derived predictions for clinical decision support (CDS).</p><p><strong>Materials and methods: </strong>An elastic-net logistic regression model developed using a national pediatric surgical registry (2012-2022) was validated in an institutional electronic health record (EHR)-derived cohort (10 450 procedures) and in a newly released, temporally distinct cohort from the same registry (301 999 procedures). We assessed discrimination, calibration, and overall performance, used a closed testing procedure to guide recalibration, and evaluated clinical utility using decision curve analysis.</p><p><strong>Results: </strong>Performance was preserved in the registry cohort, with c-statistic 0.75, Brier score 0.03, calibration slope 1.00, and observed/expected (O/E) ratio 1.01. Direct application to the institutional EHR cohort produced inflated predicted risk and poor performance, with c-statistic 0.60, Brier score 0.07, calibration slope 0.15, and O/E ratio 0.25. After omitting open-ended diagnosis-code indicators, performance improved, with c-statistic 0.70, Brier score 0.03, calibration slope 0.81, and O/E ratio 1.35. Logistic recalibration restored calibration slope and O/E ratio to 1.00. Decision curve analysis showed positive net benefit across prespecified thresholds.</p><p><strong>Discussion: </strong>These findings highlight a key challenge for clinical translation: models that perform well under internal or registry-based validation may not be directly transportable to real-world EHR environments. Predictor misalignment-differences in how model predictors were captured and measured across registry and EHR data-emerged as a key barrier to direct institutional deployment, underscoring the importance of implementation-aware external validation before clinical use.</p><p><strong>Conclusion: </strong>Addressing diagnosis-code predictor misalignment and applying parsimonious recalibration supported institutional adaptation of a pediatric SSI prediction model, providing a pragmatic pathway for translating registry-derived models into real-world clinical practice. This work sets the foundation for prospective evaluation of EHR-embedded CDS to enable individualized SSI risk estimation at the point of care.</p>","PeriodicalId":50016,"journal":{"name":"Journal of the American Medical Informatics Association","volume":" ","pages":""},"PeriodicalIF":7.1,"publicationDate":"2026-09-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149065496","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Computable phenotypes for multimorbidity measurement: development and implementation of a new electronic health record-based multimorbidity-weighted index for harmonized Observational Medical Outcomes Partnership data.","authors":"Melissa Y Wei, Ashley J Kang, Alexandra M Klomhaus, Lucia Y Chen, Douglas S Bell, David Elashoff, Chi-Hong Tseng","doi":"10.1093/jamia/ocag161","DOIUrl":"https://doi.org/10.1093/jamia/ocag161","url":null,"abstract":"<p><strong>Objective: </strong>To develop an electronic health record (EHR)-based multimorbidity-weighted index (eMWI) using computable phenotypes mapped to the Observational Medical Outcomes Partnership (OMOP) common data model and previously validated physical functioning-based condition weights.</p><p><strong>Materials and methods: </strong>We constructed eMWI from the previously validated MWI, which includes 76 chronic conditions weighted by their impact on physical functioning. For each condition, we identified validated EHR case algorithms through targeted literature search. For conditions without published algorithms, we performed chart review validation of ICD-based definitions in UCLA EHR. We translated all condition algorithms into OMOP computable phenotypes to enable use across harmonized EHR datasets. We applied eMWI to adults ≥18 years with ≥2 outpatient encounters between 2012 and 2024 in the University of California Health Data Warehouse (UCHDW), a harmonized dataset spanning 6 health systems.</p><p><strong>Results: </strong>Validated algorithms were available for 53/76 conditions. Chart review validation supported ≥1 International Classification of Diseases, Ninth and Tenth Revisions (ICD-9 and ICD-10) code definitions for 22/23 remaining conditions. We translated 76 condition definitions into OMOP phenotypes and applied eMWI in UCHDW. Final sample included 4 672 662 adults with mean ± SD age 46.3 ± 16.8 years, eMWI 5.66 ± 7.85 (range 0-91), 55.7% female, and 45.5% non-Hispanic White. eMWI increased with age and social vulnerability and captured wide distributions across population subgroups.</p><p><strong>Discussion and conclusion: </strong>eMWI enables scalable implementation of multimorbidity measurement in large harmonized EHR datasets. Its weighting to physical functioning preserves a person-centered approach while adapting disease ascertainment to rich, clinical information in EHRs. eMWI establishes a foundation for rigorous comorbidity adjustment, risk stratification, and population health research across health systems and harmonized EHR datasets.</p>","PeriodicalId":50016,"journal":{"name":"Journal of the American Medical Informatics Association","volume":" ","pages":""},"PeriodicalIF":7.1,"publicationDate":"2026-09-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149041679","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Artificial intelligence-based diagnostic decision support for primary care of older adults using collective intelligence and imitation learning.","authors":"Christopher D Streiffer, Matthew J Press, Nicholas S Bishop, Benjamin E Schmid, Alyssa Sliwa, Anuradha L Vyas, Lyle H Ungar, MaryAnne Peifer, Gary E Weissman","doi":"10.1093/jamia/ocag132","DOIUrl":"https://doi.org/10.1093/jamia/ocag132","url":null,"abstract":"<p><strong>Objective: </strong>Older adults face disproportionate risk of diagnostic errors in primary care, yet effective diagnostic decision support systems (DDSSs) remain lacking. Existing artificial intelligence (AI)-based DDSSs rely on gold-standard labels infeasible to obtain, have narrow diagnostic scope, and fail to reflect real-world uncertainty. We developed and validated a deep learning DDSS using imitation learning and collective clinician intelligence to generate diagnostic and order recommendations.</p><p><strong>Materials and methods: </strong>We trained a multi-label neural network on structured and unstructured electronic health record data from primary care encounters of adults ≥65 years old to imitate clinician decisions. Models were evaluated using discrimination, calibration, threshold-based, and composite metrics on a temporally held-out test set. Clinical validity was assessed using a randomized, blinded equivalence study comparing model-generated recommendations with observed clinician decisions.</p><p><strong>Results: </strong>The study included 707 598 primary care encounters, generating recommendations across 669 diagnoses and 1000 orders. The general model demonstrated excellent performance (micro c-statistic 0.995 [95% CI 0.995-0.996], macro c-statistic 0.896 [0.888-0.897], micro F1-score 0.904 [0.903-0.905], macro F1-score 0.384 [0.381-0.385], integrated calibration index (ICI) 0.0025 [0.0024-0.0026]). Order prediction was lower (micro c-statistic 0.812 [0.811-0.814], macro c-statistic 0.786 [0.777-0.786], micro F1-score 0.305 [0.303-0.307], macro F1-score 0.0286 [0.0277-0.0294], ICI 0.0145 [0.0141-0.0150]). In clinician-validation, diagnostic disagreements were statistically equivalent between model recommendations and observed practice while order disagreements were not.</p><p><strong>Conclusion: </strong>Imitation learning and collective clinician intelligence provide a feasible framework for generating diagnostic and ordering recommendations that reflect real-world practice without expert-adjudicated training labels. Randomized, blinded clinician evaluation pragmatically establishes preliminary clinical validity beyond traditional retrospective metrics.</p>","PeriodicalId":50016,"journal":{"name":"Journal of the American Medical Informatics Association","volume":" ","pages":""},"PeriodicalIF":7.1,"publicationDate":"2026-09-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149041713","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A co-learning framework for maternal and pediatric pharmacotherapy knowledge gap discovery.","authors":"Xiaofu Liu, Aislinn M O'Kane, Lei Wang, Aditi Shendre, Hanson Ma, Yichuan Zhang, Haoran Jiang, Chien-Wei Chiang, Syed S Zaidi, Omar A Aboshady, Gerald So, Emma M Tillman, Lindsey M Kirkpatrick, Saurabh Rahurkar, Lijun Cheng, Jiayi Ouyang, Ping Wei, Zhimo Zhu, Shangjia Li, Yirui Huang, Lingling Wang, Weidan Cao, Jianing Liu, Samuel-Richard Oteng, Andrew Goodwin, Jiezel Ann Faith Deypalubos, Shijun Zhang, Robert Bies, Sara K Quinney, Lang Li","doi":"10.1093/jamia/ocag130","DOIUrl":"https://doi.org/10.1093/jamia/ocag130","url":null,"abstract":"<p><strong>Objective: </strong>Pregnant and breastfeeding women, along with children, represent populations persistently underrepresented in clinical research, leading to uncertainty in clinical decision-making. This study investigated the landscape of pharmacology research concerning pharmacokinetics (PK), pharmaco-epidemiology (PE), and clinical trials (CT) within maternal and pediatric patient populations.</p><p><strong>Materials and methods: </strong>A pharmacology research landscape in maternal and pediatric patient populations was analyzed using a novel co-learning framework applied to 38 million PubMed abstracts. It incorporated an evolving annotation guideline for abstract labeling, uncertain sampling, annotation error calibration, and large language model training. The co-learning framework enabled human annotators and language models to iteratively learn from each other, optimizing the classification of PK/PE/CT papers. Analysis was restricted to medications prescribed for maternal and pediatric patients, focusing on publication evidence.</p><p><strong>Results: </strong>Co-learning significantly increased the F1-scores for classifying the CT, PE, and PK abstracts from (0.81, 0.84, 0.73) to (0.96, 0.94, 0.93), respectively. On the external validation set, the F1-score increased from (0.81, 0.60, 0.42) to (0.95, 0.88, 0.91) for CT, PE, and PK, respectively. The models identified 376 231 and 879 936 abstracts relevant to maternal and pediatric patients, respectively. In pregnant or postpartum women, 22%-66% prescribed drugs lacked either PK or PE/CT evidence. Among pediatric patients, 8%-40% prescribed drugs had no or weak PK or PE/CT evidence.</p><p><strong>Discussion: </strong>Co-learning is a more effective approach than active learning in obtaining feedback from human annotators for optimizing the language model. With sufficiently labeled samples, a language model such as BioBERT outperformed large language models.</p><p><strong>Conclusion: </strong>In both maternal and pediatric pharmacology research, the absence of PK evidence was notably more frequent compared to PE and CT. This significant finding calls for more pharmacokinetics studies within pediatric and maternal patient populations.</p>","PeriodicalId":50016,"journal":{"name":"Journal of the American Medical Informatics Association","volume":" ","pages":""},"PeriodicalIF":7.1,"publicationDate":"2026-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149026584","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"DiagnosticXchange: an open-source framework for evaluating safety, efficiency, and diagnostic reasoning in clinical AI systems.","authors":"Moran Sorka, Alon Gorenshtein, Hillel Abramovitch, Pannathat Soontrapa, Shahar Shelly, Dvir Aran","doi":"10.1093/jamia/ocag143","DOIUrl":"https://doi.org/10.1093/jamia/ocag143","url":null,"abstract":"<p><strong>Objective: </strong>To develop and validate an open-source evaluation framework that assesses clinical AI diagnostic systems across multiple clinically relevant dimensions (accuracy, cost, time, invasiveness, physician effort, and safety behaviors), addressing the limitations of accuracy-only benchmarks.</p><p><strong>Materials and methods: </strong>We developed DiagnosticXchange, a dynamic clinical simulation platform where AI systems interact with a simulated hospital by ordering tests, requesting imaging, and performing procedures, with each action mapped to Current Procedural Terminology (CPT) codes capturing cost, time, work relative value units, and invasiveness. We validated the framework using 8 large language models on 216 peer-reviewed cases across 19 specialties (1728 sessions). Analyses included competing-risk survival analysis, unsupervised clustering of reasoning strategies, diagnostic contribution scoring, and a pilot comparison with 14 neurologists.</p><p><strong>Results: </strong>Three systems achieved near-identical accuracy (93.5%-94.0%) yet differed significantly in cost (P <.001; 1.75-fold between the most and least expensive top-3 systems) and 2.1-fold in physician oversight. Competing-risk analysis showed the most efficient system solved 86% of cases within $5000, while the most resource-intensive required $10 000 for equivalent accuracy. Unsupervised clustering identified 3 reasoning strategies; behavioral profiles predicted resource consumption better than system identity (AIC: 4683 vs 5056). Safety analysis revealed premature diagnosis (up to 9.3%), noncontributory invasive procedures (8.7%-29.9%), and futile invasive procedures on failed cases (3.7%-18.1%). Test-retest analysis confirmed reproducible accuracy (84.6% concordance) but substantial process variability (cost CV: 86.6%). A pilot comparison captured multi-dimensional performance of 14 neurologists alongside AI systems.</p><p><strong>Conclusions: </strong>Accuracy-only evaluation is insufficient for safe clinical AI deployment. DiagnosticXchange provides an open-source, reproducible framework for multi-dimensional assessment that reveals clinically consequential differences invisible to existing benchmarks.</p>","PeriodicalId":50016,"journal":{"name":"Journal of the American Medical Informatics Association","volume":" ","pages":""},"PeriodicalIF":7.1,"publicationDate":"2026-09-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148992357","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}