Tentative renderings: Describing local data infrastructures that support the implementation and evaluation of national evaluation Initiatives

Abstract

Objective

Data journeys are a way to describe and interrogate “the life of data” (Bates et al 2010). Thus far, they have been used to clarify the mobile nature of data by visualizing the pathways made by handling and moving data. We wanted to use the data journeys method (Eleftheriou et al. 2018) to compare different data journeys by noticing repetitions, patterns, and gaps.

Methods

We conducted qualitative interviews with 43 evaluators, implementers and administrators associated with 21 clinical and training programs, called “Enterprise-Wide Initiatives” (EWIs) that are part of a national health system in the United States. We used inductive and deductive coding to identify narratives of data journeys, and then we used the “swim lane” (Collar et al 2012) format to make data journey maps based on those narratives.

Results

Unlike the actors in Eleftheriou et al. (2018)’s work, who built a data infrastructure to manage clinical data, the actors in our study built data infrastructures to evaluate clinical data. We created and compared two data journey maps that helped us explore differences in data production and management. In tracing the pathways available to the data entity of interest, and the processes through which the actors interacted with it, we noticed how the same piece of information was made to work in different ways.

Conclusions

Researchers often must build a new data infrastructures to respond to the unique needs of their evaluation work. Differing abilities lead to differences in what programs can build, and consequently what kinds of evaluation work they can support. With the goal of straightforward comparisons across different programs, a more limited focus on quantitative values, and a better description of the data journeys used by the evaluation teams, might facilitate more nuanced assessments of the evidence of complex outcomes.