Cloud Enablers For Testing Large-Scale Distributed Applications

Abstract

Testing large-scale distributed systems (also known as testing in the large) is a challenge that spreads across different technical domains and areas of expertise. Current methods and tools provide some minimal guarantees in relation to the correctness of their functional properties and have serious limitations when evaluating their extra-functional properties in realistic conditions, such as scalability, availability and performance efficiency. Cloud Testing and more specifically "testing in the cloud'' has arisen to tackle those challenges. In this new paradigm, cloud-based environment and infrastructure are used to run realistic end-to-end and/or system-level tests, collect test data and analyse them. In this paper we present a set of cloud-native services to take from the tester the responsibility of managing the resources and complementary services required to simulate realistic operational conditions and production environments. Specifically, they provide cloud testing capabilities such as logs and measurements collection from both testing jobs and system under test; test data analytics and visualization; provisioning and operation of additional services and processes to replicate realistic production ecosystems; support to scalability and diversity of underlying testing infrastructure; and replication of the operational conditions of the software under test through its instrumentation. We present the architecture of the cloud testing solution and the detailed design of each of the services; we also evaluate their relative contribution to satisfy different needs in the context of test execution.