Automated workflow regression testing for multi-tenant SaaS: integrated support in self-service configuration dashboard

Abstract

Single-instance multi-tenant SaaS applications allow tenant administrators to (extensively) customize the application according to the requirements of their organizations. In the specific case of workflow-driven applications, the SaaS provider may offer a set of pre-defined workflow activities and leave their composition to the tenant administrators. In such cases, the tenant administrator can instantiate new variants of the application without deploying new software. This effectively makes these tenant administrators part of the DevOps team, and in turn creates the need for the SaaS provider to provide them with Quality Assurance tool support. One such tool is a regression testing framework that allows them to make sure that a new version of a workflow can behave similarly as to a successful execution of a previous version. This paper highlights the potential and discusses the inherent challenges of running regression tests on workflows in the production environment of a multi-tenant SaaS application and outlines a solution in terms of architecture and automation techniques for mocking and regression detection under control of tenant administrators.