Africanus IV. The Stimela2 framework: Scalable and repeatable workflows, from local to cloud compute

Abstract

Stimela2 is a new-generation framework for developing data reduction workflows. It is designed for radio astronomy data but can be adapted for other data processing applications. Stimela2 aims at the middle ground between ease of development, human readability, and enabling robust, scalable and repeatable workflows. Stimela2 defines a YAML-based domain specific language (DSL), which represents workflows by linear, concise and intuitive YAML-format recipes. Atomic data reduction tasks (binary executables, Python functions and code, and CASA tasks) are described by YAML-format cab definitions detailing each task’s schema (inputs and outputs). The Stimela2 DSL provides a rich syntax for chaining tasks together, and encourages a high degree of modularity: recipes may be nested into other recipes, and configuration is cleanly separated from recipe logic. Tasks can be executed natively or in isolated environments using containerization technologies such as Apptainer. The container images are open-source and maintained through a companion package called cult-cargo. This enables the development of system-agnostic and repeatable workflows. Stimela2 facilitates the deployment of scalable, distributed workflows by interfacing with the Slurm scheduler and the Kubernetes API. The latter allows workflows to be readily deployed in the cloud. Previous papers in this series used Stimela2 as the underlying technology to run workflows on the AWS cloud.

This paper presents an overview of Stimela2’s design, architecture and use in the radio astronomy context.