A scoping review of outcome selection and accuracy of conclusions in complex digital health interventions for young people (2017-2023): methodological proposals for population health intervention research.

Abstract

Background: Determining the success of population health interventions often involves assessing multiple, multidimensional outcomes rather than a single one, which presents significant methodological challenges under the evidence-based medicine paradigm. This scoping review examines outcome selection, analysis, and interpretation, and the accuracy of conclusions in complex digital health interventions promoting health among adolescents and young adults (DHI-AYA).

Methods: A comprehensive search of PubMed, EMBASE, ClinicalTrials.gov, PsycINFO, and CINAHL identified DHI-AYA implemented between 2017 and 2023. Studies were categorised by methodological choice regarding outcome hierarchical position: unique primary, multiple primary, or non-hierarchised outcomes. Outcomes were further classified into effectiveness, process, or economic categories. The authors' conclusions on intervention success were compared with conclusions drawn by the research team based on the reported outcome analysis strategy. Secondly, four analytical strategies were applied to a subset of selected interventions to illustrate the impact of outcome hierarchical position and number on conclusions about intervention success.

Results: Analysis of 100 studies linked to 26 DHI-AYA identified 251 distinct outcomes: 164 effectiveness, 78 process, and 9 economic outcomes. Seven interventions were evaluated using a unique primary outcome, 10 using multiple primary outcomes, and 9 using multiple non-hierarchised outcomes. Primary and secondary outcomes were predominantly effectiveness endpoints. The research team reclassified nine interventions (35%) deemed successful by authors as non-conclusive due to statistically conflicting results across outcomes. Most interventions deemed non-conclusive by the research team were evaluated using non-hierarchised outcomes (7/10, 70%). The choice of outcome analysis strategy substantially affected conclusions on intervention success.

Conclusions: Discrepancies in intervention success assessments highlight the need for enhanced transparency, robustness, and trustworthiness in conclusion-drawing processes. In response, five methodological proposals are formulated: (1) developing core outcome sets specific to population health intervention research (PHIR), (2) collaboratively selecting multidimensional outcomes through a steering committee that accounts for stakeholder preferences and existing theoretical models, (3) exploring multi-criteria decision analysis and consensus-driven methods to transparently combine outcomes, (4) enhancing methodological reporting through intervention development and evaluation to improve scientific integrity and reproducibility, and (5) increasing PHIR expert involvement in ethics, funding, and evaluation committees to improve recognition of evidence produced in this field.

Prospero registration number: CRD42023401979.