Benchmarking the resilience of self-adaptive software systems: perspectives and challenges

Self-adaptive systems are widely recognized as the future of computer systems. Due to their dynamic and evolving nature, the characterization of self-adaptation and resilience attributes is of utmost importance, but also presents itself as a huge challenge. In fact, currently there is no practical way to characterize self-daptation capabilities, especially when comparing alternative systems concerning resilience. In this position paper we discuss the problem of resilience benchmarking of self-adaptive software systems. We identify a set of key challenges and propose a roadmap to tackle those challenges. At the same time, we present some perspectives on the development of such a benchmark, taking Autonomic Database Management Systems (ADBMS) as an illustrative case.