A Framework for Realizing Software-Defined Federations for Scientific Workflows

Abstract

Federated computing has been shown to be an effective model for harnessing the capabilities and capacities of geographically- distributed resources in order to solve large science and en- gineering problems. However, traditional High Performance Computing (HPC) based federation models can be restrictive as they present users with a pre-defined set of resources and do not allow federations to evolve in response to changing resources or application needs. As emerging application workflows and the underlying resources become increasingly dynamic and exhibit changing requirements and constraints, they cannot be easily supported by such federation models. Instead, new federation models that are capable of dynamically adapting to these emerging needs are required. In this paper, we present a programmable dynamic federation model that uses software-defined environment concepts to drive the federation process and seamlessly adapt resource compositions at runtime. The resulting software-defined federation adapts to meet both requirements and constraints set by the user, application, and/or resource providers. In this paper we present the design and prototype implementation of such software-defined federation model, and demonstrate its operation and performance through a use case where heterogeneous, geographically distributed resources are federated based on user specifications, and the federation evolves over time following the requirements and constraints defined by the user.