Wide-area Nile: a case study of a wide-area data-parallel application

Abstract

The Nile system is a distributed environment for running very large, data-intensive applications across a network of commodity workstations. These applications process data from elementary particle collisions, generated by the Cornell Electron Storage Ring, and are used by physicists of the CLEO experiment. The applications have a simple data-parallel structure, and so Nile executes them using as much parallelism as is available. Nile currently runs at any single site. It is being used by alpha testers and is scheduled for beta release in March 1998. We describe how we are adapting this local-area Nile system to allow for wide-area, multiple site interactions. In particular, we consider the two problems of scaling and of fault tolerance.