Parallel I/O: lessons learnt in the last 20 years

Abstract

Summary form only given. After these two decades, it is now a good time to go through all the done work and try to learn the important lessons all these parallel I/O initiatives have taught us. This work aims at giving this global overview. The focus is not on commercial/academic systems/prototypes, but on the concepts that lay behind them. These concepts have normally been applied at different levels, and thus, such an overview can be of interest to many people ranging from the hardware design to the application implementation. Some of the most important concepts that are discussed are, among others, data placement (RAIDs, 2D and 3D files, ...), network architectures for parallel I/O (Network attached devices, SAN, ...), parallel caching and prefetching (cooperative caching, Informed caching and prefetching, ...), and interfaces (collective I/O, data distribution interfaces, ...).