Interval Based I/O: A New Approach to Providing High Performance Parallel I/O

Abstract

Providing scalable, high-performance parallel I/O for data-intensive computations is beset by a number of difficult challenges. The most often cited difficulties include the non-contiguous I/O patterns prominent in scientific codes, the lack of support for parallel I/O optimizations in POSIX, the high cost of providing strict file consistency semantics, and the cost of accessing storage devices over a network. We believe, however, that a more fundamental problem is the legacy view of a file as a linear sequence of bytes. To address this issue, we are developing a new approach to parallel I/O that is based on what we term intervals and interval files. This paper provides an overview of the interval-IO system and a set of benchmarks demonstrating the power of this new approach.