MorphStore: A local file system for Big Data with utility-driven replication and load-adaptive access scheduling

Abstract

File system performance is critical for overall performance of Big Data workloads. Typically, Big Data file systems consist of dual layers; a local node-level file system and a global file system. This paper presents the design and implementation of MorphStore, a local file system design that significantly improves performance when accessing large files by using two key innovations. First, MorphStore uses a load-adaptive I/O access scheduling technique that dynamically achieves the benefits of striping at low load and the throughput benefits of replication at high loads. Second, MorphStore uses a utility-driven replication to maximize the utility of replication capacity by allocating replication capacity to popular read-mostly files. Experiments reveal that MorphStore achieves 8% to 12% higher throughput while using significantly less replication for workloads that access large files. If we consider the performance-capacity tradeoff of file systems built on static techniques such as JBOD, RAID-0 and RAID-1 MorphStore extends the Pareto frontier to achieve better performance at the same replication capacity.