AI-Powered Drug Classification and Indication Mapping for Pharmacoepidemiologic Studies: Prompt Development and Validation.

Abstract

Background: Pharmacoepidemiologic studies, which promote rational drug use and improve health outcomes, often require Anatomical Therapeutic Chemical Classification System (ATC) drug classification within real-world data (RWD) sources. Existing classification tools are expensive, brittle, or have restrictive terms of service, and lack context that may inform classification itself.

Objective: This study sought to establish large language models (LLMs) as an assisting technology in the drug classification task. This included developing artificial intelligence prompts that reason about drugs using RWD and showing that the resulting accuracy, efficiency, and effectiveness are favorable to alternative methods.

Methods: A prompt was constructed to classify aspirin as either an analgesic or antithrombotic and evaluated within 12,294 anonymized daily dose strings from a polychronic population residing in the United States and Canada. The patients used a smart medication dispenser called "spencer" and consented to the use of their data for research. The LLM prompt requested that the best and next-best second-level ATC code be returned, and grading was performed on a 3-point scale. After success in a pilot sample of 20, an inference sample of 200 was taken without replacement. Finite population inference was carried out on the proportion of outputs receiving 1 of the top 2 grades. As a benchmark, Google's Programmable Search Engine was used to query the drug name plus "ATC code" followed by regex-based extraction of ATC codes. All imperfect results were reviewed.

Results: The population consisted of 12,294 daily dose strings from 86.26% (2908/3371) patients residing in Canada and 13.73% (463/3371) residing in the United States. A prompt using the chain-of-thought reasoning was able to distinguish between aspirin's analgesic versus antithrombotic therapeutic uses and performed well in the pilot sample. In the inferential sample, 87.5% (175/200) were graded as perfect, 5% (10/200) had a minor issue, and 7.5% (15/200) had a major issue. The estimate of the proportion of at least mostly correct classification was 92.5% (185/200, 80% CI 90.1%-94.9%). For the search-based algorithm, 82.5% (165/200) were deemed acceptable. The chain-of-thought reasoning was most helpful with supplements (eg, folic acid) when high doses indicated antianemic preparations. The problem formulation of daily dose inputs and multiple ATC outputs was sometimes incompatible with the drug (eg, pregabalin, calcitriol, and methotrexate).

Conclusions: GPT-4o offers cost-effective drug classification from RWD without violating any terms of service. Using a chain-of-thought prompting technique, GPT-4o can reason about drug dosages that affect the class. The wide accessibility of LLMs gives every research team the ability to classify drugs at scale, a key prerequisite of pharmacoepidemiologic research.