Data Management, In-Situ Workflows and Extreme Scales

Abstract

Data-related challenges are dominating computational and data-enabled sciences and are limiting the potential impact of scientific application workflows enabled by extreme scale computing environments. While data staging and in-situ/in-transit data processing have emerged as attractive approaches for supporting these extreme scale workflows, the increasing heterogeneity of the storage hierarchy, coupled with increasing data volumes and complex and dynamic data access/exchange patterns, are impacting the effectiveness of these techniques. In this talk I will discuss these challenges and explore how autonomic runtime techniques are being explored to address them. I will then present autonomic policies as well as cross layer mechanisms that are part of DataSpaces, an extreme scale data staging service. This research is part of the DataSpaces project at the Rutgers Discovery Informatics Institute.