DAGless: Automating End-to-End Support for Serverless DAG Lifecycles With Cost-Efficient Execution

Abstract

Developing cost-efficient serverless DAGs (Directed Acyclic Graphs) is a challenging process involving (1) the non-trivial FaaSification of monolithic workflows into serverless environments, (2) profiling to determine runtime characteristics, and (3) effectively allocating tasks and resources to serverless functions. Workflows often rely on implicit dependencies (e.g., external systems, files), which introduce requirements that only surface at runtime, complicating the adaptation to serverless architectures. Managing these dependencies within serverless environments requires a holistic approach that integrates information gathered across the entire lifecycle of serverless DAGs. Unfortunately, current solutions often address only isolated parts of this process, typically requiring significant manual effort. To overcome these challenges, we introduce DAGless, a platform that automates end-to-end support for the serverless DAG lifecycle. DAGless transforms the workflow through three models, each representing a specific phase of the lifecycle, iteratively refining the serverless DAG to achieve cost-efficient execution. DAGless estimates costs of the transformed serverless DAG with a minimum accuracy of 91.07% and provides up to 8× cheaper profiling. DAGless dominates state-of-the-art cost optimizers, not only in terms of cost (16.93%), but also in terms of performance (4.27×).