Towards Building a Lightweight Key-Value Store on Parallel File System

Abstract

As data grows in number and size, big data applications begin to revolutionize the underlying storage system. On one hand, key-value store has prevailed as the back-end storage for big data applications owning to its schema-less data model, high scalability, and etc. On the other hand, parallel file system shared by multiple nodes offers large-capacity, high-throughput, as well as high-bandwidth access and is used widely in high performance computing (HPC) and cloud computing environments. In this paper, we explore the opportunity of building a lightweight key-value store that supports concurrent access over a parallel file system. The key-value store proposed relies on the sharing nature of parallel file system to provide distributed access. Instead of organizing a cluster of nodes with long running services to delegate the access, our key-value store simply embeds itself into applications and requires no long running services neither communication between nodes. Such a design not only simplifies the structure of a distributed key-value store but also avoids overhead introduced by having running services around the file system. We implemented a prototype of this system and compared it against Cassandra, a state-of-art key-value store. Preliminary results are promising.