Abstractions for supercomputing platforms to facilitate consolidation of services: poster

Abstract

Supercomputing platforms have developed over the years to maximize performance and scalability and to minimize any abstractions that could result in slowdowns and performance variability. Largescale supercomputing systems traditionally tightly integrate hardware and software stacks, for instance, light weight operating systems, vertically integrated communication layers and tightly coupled parallel file systems. Consequently, these environments do not lend themselves for portability expected for web and ISV applications as well as emerging data science workflows. This poster will present abstractions that are introduced in Piz Daint, a hybrid and heterogeneous Cray XC50 and XC40 platform with Nvidia P100 GPU devices, in order to enable Large Hadron Collider (LHC) computing grid workflows. This transition is completely transparent to the end users and existing workflows for experiments including ATLAS, CMS and LHCb. The technologies and solutions that have been exploited in creating abstractions include containerization, file system virtualization and other configuration changes. Incidentally, these abstractions have proven to be highly effective in leveraging execution of other complex workflows, for instance, Apache Spark, often with performance and productivity gains. Piz Daint hybrid and heterogeneous supercomputing platform has therefore enabled the Swiss National Supercomputing Centre (CSCS) in consolidating a range of services including computing, data analysis and visualization for its diverse and growing user communities.