Paediatric rare diseases: Can large language models assist off-label prescribing?

Aims: To evaluate the effectiveness and reliability of large language models (LLMs) in retrieving and synthesizing biomedical information to support off-label drug prescribing in paediatric rare diseases, and to compare their performance with human-authored references in terms of scientific rationale, adverse events and drug interactions.

Methods: The study reviewed 20 cases of off-label prescriptions in rare paediatric diseases using 4 LLMs (i.e., GPT-4o, Sophos-2, Claude-3, Scopus AI). The queries addressed focused on scientific rationale, adverse events and drug interactions. The performance measures encompassed sensitivity, precision, accuracy, F1-score, response quality and reference quality. A Global Performance Score integrated all measures.

Results: After evaluating 2758 references and 480 responses, a significant discrepancy was found among 4 LLMs concerning Global Performance Score (P = .001). Posthoc analysis showed that Scopus AI vs. GPT-4o comparison was significant, with GPT-4o showing higher values. Median LLM reference quality often surpassed human performance, yet variability limits conclusions regarding superiority.

Conclusions: LLMs are capable of retrieving and synthesizing biomedical information, but performance varies depending on query type and search mode. These tools speed up retrieving relevant information to assess off-label prescribing appropriateness. Despite the promise of artificial intelligence, human oversight remains critical to ensure data accuracy and reliability.