A tailored fit that doesn't fit all: the problem of threshold overfitting in diagnostic studies.

Abstract

Objectives: To critically examine the phenomenon of threshold overfitting in diagnostic accuracy research and evaluate its methodological implications through a structured review of relevant literature.

Methods: This article presents a narrative and critical review of methodological studies and reporting guidelines related to threshold selection in diagnostic test accuracy. It focuses on the misuse of post hoc thresholds, the misapplication of bias assessment tools such as QUADAS-2, and the frequent absence of independent validation. In addition to identifying these structural flaws, the article proposes a set of five concrete safeguards - ranging from transparent reporting to rigorous risk of bias classification - designed to mitigate threshold-related bias in future diagnostic studies.

Results: Thresholds are frequently derived and evaluated within the same dataset, inflating sensitivity and specificity estimates. This overfitting is seldom acknowledged and is often misclassified as low risk of bias. QUADAS-2 is frequently misapplied, with reviewers mistaking the mere presence of a threshold for proper pre-specification. The article identifies five key safeguards to mitigate this bias: (1) clear declaration of pre-specification, (2) justification of threshold choice, (3) independent validation, (4) full performance reporting across thresholds, and (5) rigorous application of bias assessment tools.

Conclusions: Threshold overfitting remains an underrecognized but methodologically critical source of bias in diagnostic accuracy studies. Addressing it requires more than awareness - it demands transparent reporting, proper validation, and stricter adherence to methodological standards.