Skitter: a DSL for distributed reactive workflows

Abstract

Writing real-time applications that react to vast amounts of incoming data is a hard problem, as the volume of incoming data implies the need for distributed execution on a cluster architecture. We envision such an application can be created as a data processing pipeline which consists of a set of generic, reactive components, which may be reused in other applications. However, there is currently no programming model or framework that enables the reactive, scalable execution of such a pipeline on a cluster architecture. Our work introduces the notion of reactive workflows, a technique that combines concepts from scientific workflows and reactive programming. Reactive workflows enable the integration of these generic components into a single workflow that can be executed on a cluster architecture in a reactive, scalable way. To deploy these reactive workflows, we introduce a domain specific language, called Skitter. Skitter enables developers to write reactive components and compose these into reactive workflows, which can be distributed over a cluster by Skitter’s runtime system.