High Performance Parallel File Objects

Abstract

High performance parallel computers are expected to solve problems involving very large data sets, often far larger than can fir in primary memory. If It0 is not performed intelligently, then the wait for I10 can become a serious bottleneck, limiting the gains from improved processor technology. This paper introduces ELFS (ExtensibLe File Systems). ELFS is a parallel, asynchronous It0 system designed to attack the I10 bottleneck. It combines recent technological advances in three areas: objectoriented systems design, latency obscuring compiler technology, and parallel disk arrays attached to parallel architectures. We present the ELFS class pfo (parallel file object), a parallel 2D-matrix class. Pfo's allow the user to: I ) specify the access pattern, e.g., row-wise, column-wise, or by sub-blocks; 2 ) partition the p f o into sub-pfos defined by subsets of the original file structure, and specify where the new sub-pfo should be located; and 3) access the file in an asynchronous and pipelined manner. Preliminary performance results are presented.