Overuse and False Restraint in Large Language Model Recommendations for Choosing Wisely Scenarios.

Abstract

Background: The Choosing Wisely initiative made recommendations where screening, testing, or treatment may be overused, with caveats around appropriate use. Clinicians are increasingly using large language models (LLMs).

Objective: To evaluate whether commercial LLMs appropriately call for restraint or action from scenarios based on Choosing Wisely recommendations.

Design: Benchmark evaluation. Fifteen Choosing Wisely recommendations were used to develop 225 scenarios (per recommendation, 10 in which restraint was appropriate and 5 in which action was appropriate). Six commercial LLMs (3 runs each) generated 4050 responses to the 225 scenarios, using a prompt from the clinician perspective.

Setting: Production application programming interfaces of OpenAI, Anthropic, and Google from February to March 2026.

Participants: No human participants were involved.

Measurements: Co-primary outcomes were the overuse rate (endorsing action when restraint was appropriate) and the false restraint rate (withholding an appropriate action).

Results: The rate of LLM-recommended overuse was 5.4% (95% CI, 0.6% to 12.2%); LLMs did not recommend overuse in 8 of 15 recommendations. Two recommendations accounted for most LLM-recommended overuse: thyroid ultrasound for an incidental subcentimeter nodule (overuse recommended in 42.8% of LLM responses) and routine electrocardiography in low-risk adults (overuse recommended in 28.3% of LLM responses). The LLMs recommended overuse in 0.6% to 3.3% of responses for the remaining 5 recommendations. False restraint was rare (0.6% [CI, 0.1% to 1.2%]), was absent for 11 of 15 recommendations, and was highest for prostate-specific antigen screening (3.3%).

Limitations: Single prompts without follow-up; a benchmark rather than real-world use; responses classified by LLM judge with physician adjudication. Findings may not extend to consumer-facing use.

Conclusion: The LLMs rarely withheld recommendations for treatment when it was appropriate. Overuse was concentrated in 2 recommendations.

Primary funding source: None.