An implicitly scalable, fully interactive multimedia storage server

We are developing a next-generation multimedia server that provides fully interactive access to tremendous amounts and varieties of real-time and non-real-time multimedia data by hundreds of simultaneous clients. Current multimedia servers are inadequate for this task, given their support of only basic multimedia data types, inherently non-interactive access semantics and/or intrinsic scaling limitations. Our solution abandons the common use of striping and object replication, and implements a random data allocation scheme across a cluster of commodity computers. This scheme provides implicit load balancing both within and among storage nodes of the cluster while supporting virtually any multimedia data type and application access pattern. This paper presents the essential background, design and implementation, and simulation studies of the storage server component of our system. Our results show that we can guarantee with high probability that an arbitrary I/O requests can be satisfied within a small delay bound while obtaining high system utilization. Although our specific application is a real-time multimedia storage server, the techniques developed can be applied to scalability in distributed systems in general.