A comparison of large language model-generated and published perioperative neurocognitive disorder recommendations: a cross-sectional web-based analysis.

IF 9.1 1区医学 Q1 ANESTHESIOLOGY

British journal of anaesthesia Pub Date : 2025-02-07 DOI:10.1016/j.bja.2025.01.001

Sarah Saxena, Odmara L Barreto Chang, Melanie Suppan, Basak Ceyda Meco, Susana Vacas, Finn Radtke, Idit Matot, Arnout Devos, Mervyn Maze, Mia Gisselbaek, Joana Berger-Estilita

{"title":"A comparison of large language model-generated and published perioperative neurocognitive disorder recommendations: a cross-sectional web-based analysis.","authors":"Sarah Saxena, Odmara L Barreto Chang, Melanie Suppan, Basak Ceyda Meco, Susana Vacas, Finn Radtke, Idit Matot, Arnout Devos, Mervyn Maze, Mia Gisselbaek, Joana Berger-Estilita","doi":"10.1016/j.bja.2025.01.001","DOIUrl":null,"url":null,"abstract":"Background: Perioperative neurocognitive disorders (PNDs) are common complications after surgery and anaesthesia, particularly in older adults, leading to increased morbidity, mortality, and healthcare costs. Therefore, major medical societies have developed recommendations for the prevention and treatment of PNDs. Our study evaluated the reliability of large language models, specifically ChatGPT-4 and Gemini, in generating recommendations for PND management and comparing them with published guidelines.Methods: We conducted an online cross-sectional web-based analysis over 48 h in June 2024. Artificial intelligence (AI)-generated recommendations were produced in six different locations across five countries (Switzerland, Belgium, Turkey, Canada, and the East and West Coasts of the USA). The English prompt 'a table of a bundle of care for perioperative neurocognitive disorders' was entered into ChatGPT-4 and Gemini, generating tables evaluated by independent reviewers. The primary outcomes were the Total Disagreement Score (TDS) and Quality Assessment of Medical Artificial Intelligence (QAMAI), which compared AI-generated recommendations with published guidelines.Results: The study generated 14 tables, with TDS and QAMAI scores showing similar results for ChatGPT-4 and Gemini (2 [1-3] vs 2 [2-3], P=0.636 and 4 [4-4] vs 4 [3-4], P=0.424, respectively). AI-generated recommendations aligned well with published guidelines, with the highest alignment observed in ChatGPT-4-generated recommendations. No complete agreement with guidelines was achieved, and lack of cited sources was a noted limitation.Conclusions: Large language models can generate perioperative neurocognitive disorder recommendations that align closely with published guidelines. However, further validation and integration of clinician feedback are required before clinical application.","PeriodicalId":9250,"journal":{"name":"British journal of anaesthesia","volume":" ","pages":""},"PeriodicalIF":9.1000,"publicationDate":"2025-02-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"British journal of anaesthesia","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1016/j.bja.2025.01.001","RegionNum":1,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"ANESTHESIOLOGY","Score":null,"Total":0}

引用次数: 0

Abstract

Background: Perioperative neurocognitive disorders (PNDs) are common complications after surgery and anaesthesia, particularly in older adults, leading to increased morbidity, mortality, and healthcare costs. Therefore, major medical societies have developed recommendations for the prevention and treatment of PNDs. Our study evaluated the reliability of large language models, specifically ChatGPT-4 and Gemini, in generating recommendations for PND management and comparing them with published guidelines.

Methods: We conducted an online cross-sectional web-based analysis over 48 h in June 2024. Artificial intelligence (AI)-generated recommendations were produced in six different locations across five countries (Switzerland, Belgium, Turkey, Canada, and the East and West Coasts of the USA). The English prompt 'a table of a bundle of care for perioperative neurocognitive disorders' was entered into ChatGPT-4 and Gemini, generating tables evaluated by independent reviewers. The primary outcomes were the Total Disagreement Score (TDS) and Quality Assessment of Medical Artificial Intelligence (QAMAI), which compared AI-generated recommendations with published guidelines.

Results: The study generated 14 tables, with TDS and QAMAI scores showing similar results for ChatGPT-4 and Gemini (2 [1-3] vs 2 [2-3], P=0.636 and 4 [4-4] vs 4 [3-4], P=0.424, respectively). AI-generated recommendations aligned well with published guidelines, with the highest alignment observed in ChatGPT-4-generated recommendations. No complete agreement with guidelines was achieved, and lack of cited sources was a noted limitation.

Conclusions: Large language models can generate perioperative neurocognitive disorder recommendations that align closely with published guidelines. However, further validation and integration of clinician feedback are required before clinical application.

查看原文本刊更多论文

求助全文

约1分钟内获得全文求助全文

来源期刊

British journal of anaesthesia 医学-麻醉学

CiteScore

13.50

自引率

7.10%

发文量

488

审稿时长

27 days

期刊介绍： The British Journal of Anaesthesia (BJA) is a prestigious publication that covers a wide range of topics in anaesthesia, critical care medicine, pain medicine, and perioperative medicine. It aims to disseminate high-impact original research, spanning fundamental, translational, and clinical sciences, as well as clinical practice, technology, education, and training. Additionally, the journal features review articles, notable case reports, correspondence, and special articles that appeal to a broader audience. The BJA is proudly associated with The Royal College of Anaesthetists, The College of Anaesthesiologists of Ireland, and The Hong Kong College of Anaesthesiologists. This partnership provides members of these esteemed institutions with access to not only the BJA but also its sister publication, BJA Education. It is essential to note that both journals maintain their editorial independence. Overall, the BJA offers a diverse and comprehensive platform for anaesthetists, critical care physicians, pain specialists, and perioperative medicine practitioners to contribute and stay updated with the latest advancements in their respective fields.