Can LLMs Predict Patient Treatment Choices? A Discrete Choice Experiment Framework.

Abstract

Objectives: This study evaluated the viability of large-language-models (LLMs), specifically GPT-4, in predicting patient health preference-consistent choices using a discrete choice experiment (DCE) framework.

Methods: Synthetic data were generated from real DCE responses by patients with a history of cancer. Analytical data included 50 synthetic patients, each answering 48 two-alternative treatment choice questions varying in expected survival, chance of long-term survival, health limitations, and out-of-pocket cost. GPT-4's predictive performance was assessed across four experiments. In Experiments 1 and 2, GPT-4 predicted 20 hold-out questions (i.e., new choice questions) leveraging 28 fixed (Experiment 1) or random selected (Experiment 2) sample questions. Experiment 3 varied the number of sample questions to examine prediction accuracy and prediction confidence. Experiment 4 evaluated how characteristics of the hold-out questions influenced prediction accuracy.

Results: GPT-4 achieved an average prediction accuracy of 70.5% (95% confidence interval [CI]: 68.3%-72.7%) in Experiment 1 and 69.9% (95% CI: 66.9%-72.9%) in Experiment 2, with greater variability when sample questions were randomized. Experiment 3 revealed a learning curve, where accuracy improved from 53% with 5 sample questions to 64% with 10, after which performance plateaued. Experiment 4 showed higher prediction accuracy for questions with more salient attribute differences.

Conclusions: GPT-4 demonstrated the ability to infer patient preferences from limited samples, achieving accuracy levels comparable to surrogate decision-makers. Its performance remained consistent across randomized input sequences and improved as the number of sample questions increased, eventually reaching a plateau where additional training yielded diminishing returns.