Bridging Network and Parallel I/O Research for Improving Data-Intensive Distributed Applications

Abstract

The rapidly evolving scene of emerging workloads poses a challenge to the High Performance Computing community in terms of communication and I/O. Significant improvements are required to keep up with the demand of high rate of data transfers, streaming services, and scientific research that deal with extremely large quantities of data, which may impede a system's performance. Networking is a key area that plays a major role in accelerating data transfers within HPC facilities. Though significant research efforts have targeted I/O optimization for storage systems, network optimization to improve the overall storage system performance has been rather overlooked by the research community. In this position paper, we aim to bridge the gap between networks and storage system optimization towards the common goal of accelerating HPC I/O and communication by revealing the various ways in which previously done network optimization research can be applied to improve I/O performance for data-intensive applications.