Spin and Hacking in Machine Learning Prediction Model Studies in Dentistry: A Systematic Review.

Abstract

Objectives: As machine learning and prediction model research grows in popularity, researchers may be tempted to exaggerate the value of their models through misleading reporting (spin) or statistical manipulation (hacking). This study assessed spin and potential hacking in machine learning-based prediction model studies in dentistry.

Methods: Six databases were searched to identify studies published up to October 12, 2024. 2780 eligible studies were identified, and 1206 AUC values were extracted from abstracts for hacking analysis. For spin assessment, 209 studies were selected and evaluated based on the SPIN-PM framework.

Results: The histogram of AUC values showed fluctuations near thresholds (0.7 and 0.8), suggesting potential AUC-hacking evidence. Spin practices were identified in 37.3% (n = 78) assessed studies, mainly through unjustified use of optimistic or positive words to describe model performance and claims of clinical applicability without external validation. Facilitators of spin were found in 39.2% (n = 82) of studies, with the most frequent being the reporting of performance measures without confidence intervals.

Conclusions: The spin practices and facilitators were prevalent, and some evidence of hacking was found. We suggest considering the 'TRIAL' (Transparency, Reporting, Integrity, Adjustment, and Learning) principles to guide machine learning prediction model studies in dentistry, thereby minimizing spin and hacking.