Measuring the aggregated impact of research: Establishing criteria for coding Translational Science Benefits Model data.

Abstract

Introduction: A promising approach to assessing research impact draws on the Translational Science Benefits Model (TSBM), an evaluation model that tracks the applied benefits of research in four domains: Clinical and Medical; Community and Public Health; Economic; and Policy and Legislative. However, standardized methods to verify TSBM benefit data, to aid in aggregating impact data within quantitative summaries, do not currently exist.

Methods: A panel of 11 topic experts participated in a modified Delphi process for establishing content and face validity of a set of criteria for verifying qualitative TSBM data. Two survey rounds were completed by panelists, with a moderated discussion in between rounds to discuss criteria not reaching consensus. Criteria with panel consensus at or above 70% in the survey rounds were confirmed as validated.

Results: Criteria fell into 9 categories: Content Relevant, Project Related, Who, Reach, What, How, Novel, Documented Evidence, and When. The Delphi process yielded 197 total criteria across the 30 benefits characterized by the TSBM (range = 5-8 criteria per benefit).

Discussion: The results of this Delphi process lay the foundation for developing a TSBM coding tool for evaluating and quantifying TSBM data. Standardizing this process will enable data aggregation, group analysis, and the comparison of research impact across contexts.