Program Evaluation Standards for Utility Facilitate Stakeholder Internalization of Evaluative Thinking in the West Virginia Clinical Translational Science Institute.

Abstract

Background: The program evaluation standards (PES) can be considered established criteria for high-quality evaluations. We emphasize PES Utility standards and evaluation capacity building as we strive for meaningful application of our work in the real world.

Purpose: We focused our methodology on understanding how stakeholders discussed utility and how their perceptions related to our evaluation work aligned with the Utility domain of the program evaluation standards.

Setting: The West Virginia Clinical Translational Science Institute (WVCTSI), a statewide multi-institutional entity for which we have conducted tracking and evaluation since 2012.

Intervention: Sustained collaborative engagement of evaluation stakeholders with the goal of increasing their utilization of evaluation products and evaluative thinking.

Research design: Case study.

Data collection and analysis: We interviewed five key stakeholders. We used themes developed from coding of interview data to inform document analyses. We used interview and document analyses to develop additional themes and illustrative examples, as well as to develop and describe a five-level evaluation uptake scale.

Findings: We describe shifts in initiation, use, and internalization of evaluative thinking by non-evaluation personnel. These shifts prompted our development and application of an evaluation uptake scale to capture increased evaluation capacity among stakeholders over time. We discuss how focus on the PES Utility standards and evaluation capacity building facilitated these shifts, and their implications for maximizing utility of evaluation activity in large, complex programmatic evaluations.