Inductive reasoning with large language models: A simulated randomized controlled trial for epilepsy

Introduction

To investigate the potential of using artificial intelligence (AI), specifically large language models (LLMs), for synthesizing information in a simulated randomized clinical trial (RCT) for an anti-seizure medication, cenobamate, demonstrating the feasibility of inductive reasoning via medical chart review.

Methods

An LLM-generated simulated RCT was conducted, featuring a placebo arm and a full-strength drug arm with a cohort of 240 patients divided 1:1. Seizure counts were simulated using a realistic seizure diary simulator. The study utilized LLMs to generate clinical notes with four neurologist writing styles and random extraneous details. A secondary LLM pipeline synthesized data from these notes. The efficacy and safety of cenobamate in seizure control were evaluated by both an LLM-based pipeline and a human reader.

Results

The AI analysis closely mirrored human analysis, demonstrating the drug's efficacy with marginal differences (<3 %) in identifying both drug efficacy and reported symptoms. The AI successfully identified the number of seizures, symptom reports, and treatment efficacy, with statistical analysis comparing the 50 %-responder rate and median percentage change between the placebo and drug arms, as well as side effect rates in each arm.

Discussion

This study highlights the potential of AI to accurately analyze noisy clinical notes to inductively produce clinical knowledge. Here, treatment effect sizes and symptom frequencies derived from unstructured simulated notes were inferred despite many distractors. The findings emphasize the relevance of AI in future clinical research, offering a scalable and efficient alternative to traditional labor-intensive data mining.