Eliciting better quality architecture evaluation scenarios: a controlled experiment on top-down vs. bottom-up

Scenarios are extensively used in software architecture evaluation. These scenarios are elicited from stakeholders using either a topdown or bottom-up approach. The former approach uses categorization schemes to focus stakeholders on developing scenarios for each required category. The latter approach uses brainstorming without any explicit categories of scenarios. It is claimed that top-down approach can result in improved quality of scenarios. However, there has been no empirical evidence on the relative effectiveness of the scenario elicitation techniques. In this paper we report on a controlled experiment with 24 subjects (postgraduate and final year undergraduate students with industry experience) in an academic context with the goal to assess the relative effectiveness of the two scenario elicitation approaches. Two groups developed scenarios to characterize quality attributes: the treatment group was given software change categories, the control group was not. The outcome variable was the quality of the scenarios produced by each participant. The average quality score for individual scenario profiles in the treatment group was significantly greater than the control group. All participants using the change categories reported that the knowledge of change categories helped them develop better quality scenarios. Our results support the claim that the provision of domainspecific software change categories helps generate better quality scenarios.