A User Study on the Usefulness of Visualization Support for Requirements Monitoring

Many requirements monitoring approaches have been proposed that check key properties of systems and their interactions at runtime. Some of these approaches also visualize monitoring results and provide details on requirements violations to end users. However, only few studies exist about the usefulness of requirements monitoring tools for practitioners, particularly regarding visualization. In this paper, we present a user study we have conducted with both industrial practitioners and researchers to assess the usefulness of visualization capabilities we have been developing for an event-based requirements monitoring tool. These capabilities allow users to monitor the status of the involved systems, to view trends and statistics, and to inspect the events and data that led to specific violations when diagnosing their root cause. We first performed a walkthrough of the tool using the cognitive dimensions of notations framework from the field of human-computer interaction. We then conducted a user study involving five software engineers of a large company from the automation software domain and four researchers. Using the tool's visualization capabilities all subjects succeeded in monitoring a real-world automation system and in diagnosing violations. Subjects regarded the visualization capabilities as essential for understanding the behavior of a complex system. Based on the study results we derive implications, opportunities, and risks of using visualization in requirements monitoring tools.