Bernardo Sousa-Pinto, Rafael José Vieira, Manuel Marques-Cruz, Antonio Bognanni, Sara Gil-Mata, Slava Jankin, Joana Amaro, Liliane Pinheiro, Marta Mota, Mattia Giovannini, Leticia de Las Vecillas, Ana Margarida Pereira, Justyna Lityńska, Boleslaw Samolinski, Jonathan Bernstein, Mark Dykewicz, Martin Hofmann-Apitius, Marc Jacobs, Nikolaos Papadopoulos, Sian Williams, Torsten Zuberbier, João A Fonseca, Ricardo Cruz-Correia, Jean Bousquet, Holger J Schünemann
{"title":"Artificial Intelligence-Supported Development of Health Guideline Questions.","authors":"Bernardo Sousa-Pinto, Rafael José Vieira, Manuel Marques-Cruz, Antonio Bognanni, Sara Gil-Mata, Slava Jankin, Joana Amaro, Liliane Pinheiro, Marta Mota, Mattia Giovannini, Leticia de Las Vecillas, Ana Margarida Pereira, Justyna Lityńska, Boleslaw Samolinski, Jonathan Bernstein, Mark Dykewicz, Martin Hofmann-Apitius, Marc Jacobs, Nikolaos Papadopoulos, Sian Williams, Torsten Zuberbier, João A Fonseca, Ricardo Cruz-Correia, Jean Bousquet, Holger J Schünemann","doi":"10.7326/ANNALS-24-00363","DOIUrl":null,"url":null,"abstract":"<p><strong>Background: </strong>Guideline questions are typically proposed by experts.</p><p><strong>Objective: </strong>To assess how large language models (LLMs) can support the development of guideline questions, providing insights on approaches and lessons learned.</p><p><strong>Design: </strong>Two approaches for guideline question generation were assessed: 1) identification of questions conveyed by online search queries and 2) direct generation of guideline questions by LLMs. For the former, the researchers retrieved popular queries on allergic rhinitis using Google Trends (GT) and identified those conveying questions using both manual and LLM-based methods. They then manually structured as guideline questions the queries that conveyed relevant questions. For the second approach, they tasked an LLM with proposing guideline questions, assuming the role of either a patient or a clinician.</p><p><strong>Setting: </strong>Allergic Rhinitis and its Impact on Asthma (ARIA) 2024 guidelines.</p><p><strong>Participants: </strong>None.</p><p><strong>Measurements: </strong>Frequency of relevant questions generated.</p><p><strong>Results: </strong>The authors retrieved 3975 unique queries using GT. From these, they identified 37 questions, of which 22 had not been previously posed by guideline panel members and 2 were eventually prioritized by the panel. Direct interactions with LLMs resulted in the generation of 22 unique relevant questions (11 not previously suggested by panel members), and 4 were eventually prioritized by the panel. In total, 6 of 39 final questions prioritized for the 2024 ARIA guidelines were not initially thought of by the panel. The researchers provide a set of practical insights on the implementation of their approaches based on the lessons learned.</p><p><strong>Limitation: </strong>Single case study (ARIA guidelines).</p><p><strong>Conclusion: </strong>Approaches using LLMs can support the development of guideline questions, complementing traditional methods and potentially augmenting questions prioritized by guideline panels.</p><p><strong>Primary funding source: </strong>Fraunhofer Cluster of Excellence for Immune-Mediated Diseases.</p>","PeriodicalId":7932,"journal":{"name":"Annals of Internal Medicine","volume":" ","pages":"1518-1529"},"PeriodicalIF":19.6000,"publicationDate":"2024-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Annals of Internal Medicine","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.7326/ANNALS-24-00363","RegionNum":1,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2024/9/24 0:00:00","PubModel":"Epub","JCR":"Q1","JCRName":"MEDICINE, GENERAL & INTERNAL","Score":null,"Total":0}
引用次数: 0
Abstract
Background: Guideline questions are typically proposed by experts.
Objective: To assess how large language models (LLMs) can support the development of guideline questions, providing insights on approaches and lessons learned.
Design: Two approaches for guideline question generation were assessed: 1) identification of questions conveyed by online search queries and 2) direct generation of guideline questions by LLMs. For the former, the researchers retrieved popular queries on allergic rhinitis using Google Trends (GT) and identified those conveying questions using both manual and LLM-based methods. They then manually structured as guideline questions the queries that conveyed relevant questions. For the second approach, they tasked an LLM with proposing guideline questions, assuming the role of either a patient or a clinician.
Setting: Allergic Rhinitis and its Impact on Asthma (ARIA) 2024 guidelines.
Participants: None.
Measurements: Frequency of relevant questions generated.
Results: The authors retrieved 3975 unique queries using GT. From these, they identified 37 questions, of which 22 had not been previously posed by guideline panel members and 2 were eventually prioritized by the panel. Direct interactions with LLMs resulted in the generation of 22 unique relevant questions (11 not previously suggested by panel members), and 4 were eventually prioritized by the panel. In total, 6 of 39 final questions prioritized for the 2024 ARIA guidelines were not initially thought of by the panel. The researchers provide a set of practical insights on the implementation of their approaches based on the lessons learned.
Limitation: Single case study (ARIA guidelines).
Conclusion: Approaches using LLMs can support the development of guideline questions, complementing traditional methods and potentially augmenting questions prioritized by guideline panels.
Primary funding source: Fraunhofer Cluster of Excellence for Immune-Mediated Diseases.
期刊介绍：
Established in 1927 by the American College of Physicians (ACP), Annals of Internal Medicine is the premier internal medicine journal. Annals of Internal Medicine’s mission is to promote excellence in medicine, enable physicians and other health care professionals to be well informed members of the medical community and society, advance standards in the conduct and reporting of medical research, and contribute to improving the health of people worldwide. To achieve this mission, the journal publishes a wide variety of original research, review articles, practice guidelines, and commentary relevant to clinical practice, health care delivery, public health, health care policy, medical education, ethics, and research methodology. In addition, the journal publishes personal narratives that convey the feeling and the art of medicine.
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号
