A splash of color: a dual dive into the effects of EVO on decision-making with goal models

Recent approaches have investigated assisting users in making early trade-off decisions when the future evolution of project elements is uncertain. These approaches have demonstrated promise in their analytical capabilities; yet, stakeholders have expressed concerns about the readability of the models and resulting analysis, which builds upon Tropos. Tropos is based on formal semantics enabling automated analysis; however, this creates a problem of interpreting evidence pairs. The aim of our broader research project is to improve the process of model comprehension and decision-making by improving how analysts interpret and make decisions. We extend and evaluate a prior approach, called EVO, which uses color to visualize evidence pairs. In this article, we explore the effectiveness of EVO with and without the impacts of tooling through a two-phased empirical study. All subjects in both phases were untrained modelers, given training at study time. First, we conduct an experiment to measure any effect of using colors to represent evidence pairs. Second, we explore how subjects engage in decision-making activities (with or without color) through a user study. We find that the EVO color visualization significantly improves the speed of model comprehension and is perceived as helpful by study subjects.