Associating architectural issues with quality attributes: a survey on expert agreement

Abstract

The architectural decision-making process is a complex and crucial endeavor in companies that develop large and distributed software systems. In this process, choosing and evaluating a solution for each architectural issue depends on decision drivers. The drivers are mainly the business factors (e.g., cost, time-to-market, etc.) and software quality attributes (e.g., security, adaptability, etc.). This paper examines whether there is agreement among experts in associating (i.e., relating) architectural issues with relevant quality attributes. We conducted a survey with 37 experts from several industrial domains who, at least once a month, make one or more architectural decisions. The results show there is poor agreement among these experts in identifying and scoring relevant quality attributes for each architectural issue. Poor agreement implies that the associating task is subjective, and that experts inconsistently define and interpret the relevance of various quality attributes for a given architectural issue that may hurt the sustainability of their architectural decisions. This paper suggests that practitioners in their decision-making should employ approaches that are more systematic. The approaches should be supported by methods and tools designed to diminish the biases of intuitive, experience-based approaches of associating architectural issues with quality attributes.