ScaDL 2022 Invited Talk 2: AI/ML Pipelines using CodeFlare

Abstract

Pipelines have become a ubiquitous construct in machine learning spanning tasks ranging from data cleaning and preprocessing, training foundational models, model optimization and transfer learning and low latency inferencing. While the many pipeline construct has existed for many years (e.g., SciKit learn pipelines, Spark pipelines), this talk will focus on a process calculus style definition of pipeline - called CodeFlare pipelines - that makes it readily amenable to scaling complex AI/ML workflows on a commodity cluster. CodeFlare pipelines not only enable data scientists to introduce compute, data and multi-stage parallelism using simple annotations on the pipeline graph, but also operationalize them on a hybrid cloud platform (Red Hat OpenShift), thereby making the solution deployable just about anywhere and leverage the benefits of serverless computing. This talk will cover a basic realization of CodeFlare pipelines on the Ray platform (1.7.0 release) that has shown near linear scalability for transfer learning and inferencing on foundational models.