Compactor: Optimization Framework at Staging I/O Nodes

Abstract

Data-intensive applications are largely influenced by I/O performance on HPC systems and the scalability of such applications to exascale primarily depends on the scalability of the I/O performance on HPC systems in the future. To mitigate the I/O performance, recent HPC systems make use of staging nodes to delegate I/O requests and in-situ data analysis. In this paper, we present the Compactor framework and also present three optimizations to improve I/O performance at the data staging nodes. The first optimization performs collective buffering across requests from multiple processes. In the second optimization, we present a way to steal writes to service read request at the staging node. Finally, we also provide a way to "morph" write requests from the same process. All optimizations were implemented as a part of the Exascale FastForward I/O stack. We evaluated the optimizations over a PVFS2 file system using a micro-benchmark and Flash I/O benchmark. Our results indicate significant performance benefits with our framework. In the best case the compactor is able to provide up to 70% improvement in performance.