Do Search-Based Approaches Improve the Design of Self-Adaptive Systems? A Controlled Experiment

Endowing software systems with self-adaptation capabilities has shown to be quite effective in coping with uncertain and dynamic operational environments as well as managing the complexity generated by non-functional requirements. Nowadays, a large number of approaches tackle the issue of enabling self-adaptive behavior from different perspectives and under diverse assumptions, making it harder for architects to make judicious decisions about design alternatives and quality attributes tradeoffs. It has currently been claimed that search-based software design approaches may improve the quality of resulting artifacts and the productivity of design processes, as a consequence of promoting a more comprehensive and systematic representation of design knowledge and preventing design bias and false intuition. To the best of our knowledge, no controlled experiments have been performed to provide sound evidence of such claim in the self-adaptive systems domain. In this paper, we report the results of a quasi-experiment performed with 24 students of a graduate program in Distributed and Ubiquitous Computing. The experiment evaluated the design of self-adaptive systems using a search-based approach, in contrast to the use of a style-based non-automated approach. The results show that search-based approaches can improve the effectiveness of resulting architectures and reduce design complexity. We found no evidence regarding the method's potential for leveraging the acquisition of distilled design knowledge by novice software architects.