HDFSbench: Understanding the Efficiency and Bottlenecks of Cloud File Systems

Abstract

We have conducted intensive experiments on an in-house Hadoop cluster using HDFSbench (a file system benchmark tool we build for HDFS). Our experimental results provide valuable insights into the performance characteristics (e.g., general efficiency and potential bottlenecks) of cloud file systems for different application usages (e.g., MapReduce and Bigtable access patterns), and on how these traits change with new storage technologies (e.g., SSD vs. HDD).