A distributed memory out-of-core method on HPC clusters and its application to quantum chemistry applications

Out-of-core methods, which repeatedly offload data to disk in order to overcome local on-node memory constraints are encountered in a range of scientific computing disciplines, including quantum chemistry. Unfortunately, these methods do not often map nicely onto global parallel file systems employed on modern HPC clusters and can overwhelm even the most capable of file systems causing unacceptably low application performance (while also degrading I/O performance for all system users). To address this bottleneck and explore more efficient use of HPC clusters for a quantum chemistry application, CFOUR, a new MPI-based utility has been developed to support out-of-core methods on distributed memory systems. This MPI Ocore utility leverages the high-speed interconnect available on HPC clusters to offload and retrieve out-of-core records to one or more remote memory storage pools, avoiding excessive I/O transactions on local or global file systems. In this paper, we present an overview of the Ocore implementation, it's direct application within a large quantum chemistry application, and micro-benchmark and application performance results from an HPC cluster interconnected with InfiniBand.