The Fallacy of Patient-Reported Outcome Measures: Why They Often Fail to Detect Clinically Important Differences.

Background: Patient-reported outcome measures are increasingly common in orthopaedic research yet routinely fail to detect clinically meaningful differences in clinical trials. We assessed if orthopaedic studies are more likely to detect clinically important differences with a binary outcome, such as nonunion surgery, or a continuous patient-reported outcome sensitive to important clinical differences.

Methods: We constructed a hypothetical clinical trial comparing 2 treatments for tibial shaft fractures. We assumed the true nonunion rate was 10% in the treatment group and 20% in the control group. Based on recent publications, we assumed nonunion patients had a mean Patient-Reported Outcomes Measurement Information System (PROMIS) physical function score of 40 (standard deviation [SD], 10) and united patients had a mean score of 50 (SD, 10), a difference that is double the minimum clinically important difference. Based on these expected distributions, we simulated 10,000 trials with a sample size of 400 patients to assess the probability of detecting a significant difference between treatment groups at p < 0.05. The trial results were analyzed using the binary outcome of nonunion surgery and PROMIS physical function.

Results: Of the 10,000 simulated trials with a 400-patient sample, the observed difference in nonunion surgery between the treatment groups was statistically significant 80.9% (95% confidence interval [CI], 79.2%- 82.6%) of the time. By contrast, the 400-patient sample detected a statistically significant difference between groups in PROMIS physical function only 4.2% (95% CI, 3.8%-4.6%) of the time (difference, 76.7%; 95% CI, 75.8%-77.6%; p < 0.001).

Conclusion: Even if patient-reported outcome measures are sensitive to important clinical events, most studies will fail to detect statistically significant differences between the treatment groups when the important clinical outcome occurs in a minority of the overall sample. Orthopaedic surgeons and scientists should be aware of this limitation when designing and appraising studies that compare treatments with patient-reported outcomes.

Level of evidence: Not applicable.