Experiment as Code - An Automated Testbed for Cloud Scaling and Operations

Abstract

Cloud Computing has been widely adopted in industry, administrations, as well as research. At the same time, development and testing of new technologies is troubled by inconsistent environments. Exchanging modules of a system during research becomes a major obstacle. As a result, baselines for comparisons are lacking, and evaluating new developments may lead to heavy time cost. Big industrial players typically circumvent this problem by relying on the scale of their datacenters, experiments, and problems, which makes them inherently interesting.We propose a softwarised testbed that abstracts away inevitable hardware differences. Its design closely follows the established philosophy to separate workloads and declare expressive interfaces between them. The result is a coherent testbed, consisting of interchangeable, but state of the art, components. It establishes Experiment-as-Code, where an experiment is declared to load generators programmatically. In sum, the testbed yields reproducible results. We verify networking, computational and interacting components by reproducing known results. We illustrate one use case of scaling under uncertainty the applicability for research. The whole system is hosted on-premise, with minimal hardware requirements, which enables especially academia to contribute. The low-effort in setup also allows handover to students and staff with the goal to ensure knowledge transfer.