Poor sample size reporting quality and insufficient sample size in economic evaluations conducted alongside pragmatic trials: a cross-sectional survey

Abstract

Objectives

Economic evaluations based on well-designed and -conducted pragmatic randomized controlled trials (pRCTs) can provide valuable evidence on the cost-effectiveness of interventions, enhancing the relevance and applicability of findings to healthcare decision-making. However, economic evaluation outcomes are seldom taken into consideration during the process of sample size calculation in pragmatic trials. The reporting quality of sample size and information on its calculation in economic evaluations that are well-suited to pRCTs remain unknown. This study aims to assess the reporting quality of sample size and estimate the power values of economic evaluations in pRCTs.

Study Design and Setting

We conducted a cross-sectional survey using data of pRCTs available from PubMed and OVID from 1 January 2010 to 24 April 2022. Two groups of independent reviewers identified articles; three groups of reviewers each extracted the data. Descriptive statistics presented the general characteristics of included studies. Statistical power analyses were performed on clinical and economic outcomes with sufficient data.

Results

The electronic search identified 715 studies and 152 met the inclusion criteria. Of these, 26 were available for power analysis. Only 9 out of 152 trials (5.9%) considered economic outcomes when estimating sample size, and only one adjusted the sample size accordingly. Power values for trial-based economic evaluations and clinical trials ranged from 2.56% to 100% and 3.21%–100%, respectively. Regardless of the perspectives, in 14 out of the 26 studies (53.8%), the power values of economic evaluations for quality-adjusted life years (QALYs) were lower than those of clinical trials for primary endpoints (PEs). In 11 out of the 24 (45.8%) and in 8 out of the 13 (61.5%) studies, power values of economic evaluations for QALYs were lower than those of clinical trials for PEs from the healthcare and societal perspectives, respectively. Power values of economic evaluations for non-QALYs from the healthcare and societal perspectives were potentially higher than those of clinical trials in 3 out of the 4 studies (75%). The power values for economic outcomes in Q1 were not higher than those for other journal impact factor quartile categories.

Conclusion

Theoretically, pragmatic trials with concurrent economic evaluations can provide real-world evidence for healthcare decision makers. However, in pRCT-based economic evaluations, limited consideration, and inadequate reporting of sample-size calculations for economic outcomes could negatively affect the results’ reliability and generalisability. We thus recommend that future pragmatic trials with economic evaluations should report how sample sizes are determined or adjusted based on the economic outcomes in their protocols to enhance their transparency and evidence quality.