Performance Optimization under Small Files Intensive Workloads in BWFS

Abstract

We have designed and implemented the Blue Whale File System (BWFS), a scalable distributed file system for large distributed data-intensive applications. With many of the features as previous distributed file systems, BWFS has successfully met our storage needs and is widely deployed within many fields. Although excellent for high-bandwidth access to large files, BWFS's out-of-band data transfer mode provides low efficiency under small files intensive workloads. In order to improve the overall performance of the file system, we propose a novel data transfer scheme. In such novel scheme, BWFS transfers data with the hybrid data transfer policy that small files are transferred with in-band mode while large files are transferred with out-of-band mode. The prototype design and implementation is described and the various experiments are presented to demonstrate that the significant performance benefits of our prototype implementation under the small files intensive workloads. For small files intensive applications, BWFS can achieve significantly higher throughput which increases by 60%.