IndigoStore: Latency Optimized Distributed Storage Backend for Cloud-Scale Block Storage

The major usage of a distributed block storage integrated with a cloud computing platform is to provide the storage for VM (virtual machine) instances. Traditional desktop and server applications tend to be written with small I/O being dominant, and in limited parallelism. Hence the performance of block storage serving these applications migrated to cloud is largely determined by latency of small I/O. This paper presents IndigoStore, an optimized Ceph backend to implement cloud-scale block storage that provides virtual disks for cloud VMs. The design of IndigoStore aims to optimize Ceph BlueStore backend, the state-of-the-art distributed storage backend, to reduce both average and tail latency of small I/O, meanwhile not waste disk bandwidth serving large I/O. We use both microbenchmarks and our production workloads to demonstrate that IndigoStore achieves 29%∼44 % lower average latency, and up to 1.23×lower 99.99th percentile tail latency than BlueStore, without any notable negative effects on other performance metrics.