Dependability Stress Testing of Cloud Infrastructures

Modern distributed systems have reached a level of complexity where software bugs and hardware failures are no longer exceptional, but a permanent operational threat. This holds especially for cloud infrastructures, which need to deliver resources to their customers under well-defined service-level agreements. Dependability need to be assessed carefully. This article presents a structured approach for dependability stress testing in a cloud infrastructure. We automatically determine and inject the maximum amount of simultaneous non-fatal errors in different variations. This puts the existing resiliency mechanisms under heavy load, so that they are tested for their effectiveness in corner cases. The starting point is a failure space dependability model of the system. It includes the notion of fault tolerance dependencies, which encode fault-triggering relations between different software layers. From the model, our deterministic algorithm automatically derives fault injection campaigns that maximize dependability stress. The article demonstrates the feasibility of the approach with an assessment of a fault tolerant OpenStack cloud infrastructure deployment.