From Evidence to Recommendations With Large Language Models: A Feasibility Study

Background

Formulating evidene-based recommendations for practice guidelines is a complex process that requires substantial expertise. Artificial intelligence (AI) is promising in accelerating the guideline development process. This study evaluates the feasibility of leveraging five large language models (LLMs)—ChatGPT-3.5, Claude-3 sonnet, Bard, ChatGLM-4, Kimi chat—to generate recommendations based on structured evidence, assesses their concordance, and explores the potential for AI.

Methods

The general and specific prompts were drafted and validated. We searched PubMed to include evidence-based guidelines related to health and lifestyle. We randomly selected one recommendation from every included guideline as the sample and extracted the evidence base supporting the selected recommendations. The prompts and evidence were fed into five LLMs to generate structured recommendations.

Results

ChatGPT-3.5 demonstrated the highest proficiency in comprehensively extracting and synthesizing evidence to formulate novel insights. Bard consistently adhered to existing guideline principles, aligning its algorithm with these tenets. Claude generated fewer topical recommendations, focusing instead on evidence analysis and mitigating irrelevant information. ChatGLM-4 exhibited a balanced approach, combining evidence extraction with adherence to guideline principles. Kimi showed potential in generating concise and targeted recommendations. Among the six generated recommendations, average consistency ranged from 50% to 91.7%.

Conclusion

The findings of this study suggest that LLMs hold immense potential in accelerating the formulation of evidence-based recommendations. LLMs can rapidly and comprehensively extract and synthesize relevant information from structured evidence, generating recommendations that align with the available evidence.