Gamifying and evaluating problem structuring: A card game workshop for generating decision objectives

Serious games, gamification, or game-based interventions are increasingly used as tools to aid participatory decision-making processes, but their evaluation is often not very rigorous. Therefore, it is still unclear whether game-based interventions are really beneficial. We focused on the following overarching question: how effective are game-based interventions specifically designed to support decision-making processes. We used an illustrative case to reflect on this question. Using a published framework proposing that design processes of game-based interventions and their evaluation are intertwined, we designed simultaneously (1) a game-based intervention, specifically a card game and a workshop structure in which this card game is to be used, and (2) its evaluation procedure, formulating evaluation questions and proposing measure instruments based on the literature. We pre-tested the evaluation procedure in a small pilot study with 10 students. We illustrate the use of the design framework for an intervention to generate objectives in a decision-making process about sustainable wastewater management. Through our illustrative case, we identify future research opportunities about designing game-based interventions and evaluating their effectiveness. We found that it is possible to address the dual challenge of game-based interventions for participatory decision-making processes: (1) designing an informative and engaging game-based intervention without telling participants what to think and (2) designing a tailored evaluation procedure. Designing the game-based intervention and its evaluation simultaneously is valuable, because both are strongly intertwined. However, conducting the evaluation is demanding and requires the collaborative efforts of scientists, including across disciplinary boundaries. For instance, the data collection effort could be distributed among different research groups to increase sample size. This would allow including control treatment(s) and covering the variation span of the confounding factors more broadly. All material is made openly available to foster collaborative future research.