Machine Learning-Assisted Health Economics and Policy Reviews: A Comparative Assessment.

Abstract

Introduction: The growth of scientific literature in health economics and policy represents a challenge for researchers conducting literature reviews. This study explores the adoption of a machine learning (ML) tool to enhance title and abstract screening. By retrospectively assessing its performance against the manual screening of a recent scoping review, we aimed to evaluate its reliability and potential for streamlining future reviews.

Methods: ASReview was utilised in 'Simulation Mode' to evaluate the percentage of relevant records found (RRF) during title/abstract screening. A dataset of 10,246 unique records from three databases was considered, with 135 relevant records labelled. Performance was assessed across three scenarios with varying levels of prior knowledge (PK) (i.e., 5, 10, or 15 records), using both sampling and heuristic stopping criteria, with 100 simulations conducted for each scenario.

Results: The ML tool demonstrated strong performance in facilitating the screening process. Using the sampling criterion, median RRF values stabilised at 97% with 25% of the sample screened, saving reviewers approximately 32 working days. The heuristic criterion showed similar median values, but greater variability due to premature conclusions upon reaching the threshold. While higher PK levels improved early-stage performance, the ML tool's accuracy stabilised as screening progressed, even with minimal PK.

Conclusions: This study highlights the potential of ML tools to enhance the efficiency of title and abstract screening in health economics and policy literature reviews. To fully realise this potential, it is essential for regulatory bodies to establish comprehensive guidelines that ensure ML-assisted reviews uphold rigorous evidence quality standards, thereby enhancing their integrity and reliability.