2060-LB: Benchmarking AI in Type 1 Diabetes Management—How Well Do Generative AI Platforms Follow ADA SOC?

IF 7.5 1区医学 Q1 ENDOCRINOLOGY & METABOLISM

Diabetes Pub Date : 2025-06-13 DOI:10.2337/db25-2060-lb

GEANNA JADA MIRANDA, SHIRLEY M.T. WONG, CLIPPER F. YOUNG

{"title":"2060-LB: Benchmarking AI in Type 1 Diabetes Management—How Well Do Generative AI Platforms Follow ADA SOC?","authors":"GEANNA JADA MIRANDA, SHIRLEY M.T. WONG, CLIPPER F. YOUNG","doi":"10.2337/db25-2060-lb","DOIUrl":null,"url":null,"abstract":"Introduction and Objective: With the rise of generative AI in healthcare, this study evaluates the internal consistency (within platforms) and external consensus (among the platforms) of diabetes care following the ADA SOC generated by AI platforms (OpenAI ChatGPT-4, Google Gemini, Copilot, Perplexity) and compare their responses with clinicians’ notes. Methods: Eight de-identified clinical cases with type 1 diabetes were extracted from a database and edited for clarity. Cases were input into the platforms, generating seven rounds of responses per case per platform. Outputs were analyzed across seven themes: Glycemic management; Lifestyle recommendations; Patient education; Psychosocial considerations; Preventative screenings and immunizations; Patient-specific considerations; and Social determinants of health. Results: Conclusion: The AI platforms showed various levels of internal consistency in their responses, with the highest consistency in CGM patient education. For external consensus among all platforms, the theme with the closest scores was CGM patient education; however, the theme with the highest Jaccard Similarity Score was Discussing Insulin Treatments. AI-generated guidance diverged markedly from clinicians’ recommendations. While more research is needed, these findings emphasize the potential of AI in supplementing diabetes care and highlight the need for human oversight to ensure comprehensive, patient-centered management. Disclosure G. Miranda: None. S.M. Wong: None. C.F. Young: Consultant; Sanofi.","PeriodicalId":11376,"journal":{"name":"Diabetes","volume":"4 1","pages":""},"PeriodicalIF":7.5000,"publicationDate":"2025-06-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Diabetes","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.2337/db25-2060-lb","RegionNum":1,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"ENDOCRINOLOGY & METABOLISM","Score":null,"Total":0}

引用次数: 0

Abstract

Introduction and Objective: With the rise of generative AI in healthcare, this study evaluates the internal consistency (within platforms) and external consensus (among the platforms) of diabetes care following the ADA SOC generated by AI platforms (OpenAI ChatGPT-4, Google Gemini, Copilot, Perplexity) and compare their responses with clinicians’ notes. Methods: Eight de-identified clinical cases with type 1 diabetes were extracted from a database and edited for clarity. Cases were input into the platforms, generating seven rounds of responses per case per platform. Outputs were analyzed across seven themes: Glycemic management; Lifestyle recommendations; Patient education; Psychosocial considerations; Preventative screenings and immunizations; Patient-specific considerations; and Social determinants of health. Results: Conclusion: The AI platforms showed various levels of internal consistency in their responses, with the highest consistency in CGM patient education. For external consensus among all platforms, the theme with the closest scores was CGM patient education; however, the theme with the highest Jaccard Similarity Score was Discussing Insulin Treatments. AI-generated guidance diverged markedly from clinicians’ recommendations. While more research is needed, these findings emphasize the potential of AI in supplementing diabetes care and highlight the need for human oversight to ensure comprehensive, patient-centered management. Disclosure G. Miranda: None. S.M. Wong: None. C.F. Young: Consultant; Sanofi.

查看原文本刊更多论文

2060-LB: 1型糖尿病管理中的基准AI -生成式AI平台遵循ADA SOC的效果如何？

随着生成式人工智能在医疗保健领域的兴起，本研究评估了人工智能平台（OpenAI ChatGPT-4、谷歌Gemini、Copilot、Perplexity）生成的ADA SOC对糖尿病护理的内部一致性（平台内部）和外部一致性（平台之间），并将其反应与临床医生的记录进行了比较。方法：从数据库中提取8例未确定的1型糖尿病临床病例，并对其进行编辑。案例被输入到平台中，每个平台每个案例产生七轮响应。分析了七个主题的产出：血糖管理；生活方式的建议;病人教育;社会心理因素;预防性筛查和免疫接种；针对病人的注意事项;健康的社会决定因素。结果：结论：人工智能平台的响应具有不同程度的内部一致性，其中CGM患者教育的一致性最高。在各平台的外部共识中，得分最接近的主题是CGM患者教育；然而，Jaccard相似性得分最高的主题是讨论胰岛素治疗。人工智能生成的指南与临床医生的建议明显不同。虽然还需要更多的研究，但这些发现强调了人工智能在补充糖尿病护理方面的潜力，并强调了人类监督的必要性，以确保全面的、以患者为中心的管理。米兰达：没有。黄：没有。C.F. Young：顾问；赛诺菲。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Diabetes 医学-内分泌学与代谢

CiteScore

12.50

自引率

2.60%

发文量

1968

审稿时长

1 months

期刊介绍： Diabetes is a scientific journal that publishes original research exploring the physiological and pathophysiological aspects of diabetes mellitus. We encourage submissions of manuscripts pertaining to laboratory, animal, or human research, covering a wide range of topics. Our primary focus is on investigative reports investigating various aspects such as the development and progression of diabetes, along with its associated complications. We also welcome studies delving into normal and pathological pancreatic islet function and intermediary metabolism, as well as exploring the mechanisms of drug and hormone action from a pharmacological perspective. Additionally, we encourage submissions that delve into the biochemical and molecular aspects of both normal and abnormal biological processes. However, it is important to note that we do not publish studies relating to diabetes education or the application of accepted therapeutic and diagnostic approaches to patients with diabetes mellitus. Our aim is to provide a platform for research that contributes to advancing our understanding of the underlying mechanisms and processes of diabetes.