Automatically encapsulating HPC best practices into data transfers

Abstract

This paper presents the Shift automated transfer tool and the mechanisms it employs to achieve better performance while preserving the stability of HPC environments. Shift encapsulates best practices understood by domain experts during transfers so that scientists can focus on their science without the need to study file transports, resource management, and file systems as well. Shift understands how to utilize the variety of transports that might be deployed throughout a widely distributed user base, how to maximize the performance achievable by each, and the scenarios in which each is most effective. Shift understands which resources are available in a particular HPC environment and how to utilize them for significant performance increases while preventing resource exhaustion. Finally, Shift understands the file systems to which and from which files may be transferred and the nuances to their use that affect performance and stability behind the scenes.