Feeding the Beast: Getting Data into Big Systems

Abstract

The distance between compute and data continues to grow and despite being declared "dead" a decade ago, the interfaces of the serial world (Posix) are still with us. Is it time to move on and to what? How can we give applications the desired flexibility, consistency and volume of data in large systems composed of thousands or millions of failing parts? The group Brent runs at Intel, the High Performance Data Division, has been shepherding the Lustre file system while working on the riddle of "what now" for big systems when it comes to feeding them data.