NM2H: Design and Implementation of NoSQL Extension for HDFS Metadata Management

Abstract

As a distributed MapReduce framework, Hadoop has been widely adopted in big data processing, in which HDFS (Hadoop Distributed File System) is mostly used for data storage. Though the single master architecture of HDFS simplifies the design and implementation, it suffers from issues such as SPOF (Single Point Of Failure) and scalability, which further may become performance bottleneck. To address these problems, this paper proposes NM2H, a NoSQL based metadata management approach for HDFS. NM2H separates the storage and query of metadata in contrast to the traditional architecture which mixed them up, and manages to keep the interfaces among the metadata service, Data Nodes and clients unchanged through a novel mapping mechanism between the original metadata structures to NoSQL documents. Therefore, the new approach can not only take advantages of NoSQL's better scalability and fault tolerance, but also deliver transparency to client applications, in which way existing programs can run on the new architecture without any modification. The prototype of NM2H was designed and implemented with widely adopted NoSQL system MongoDB. Extensive performance evaluation was conducted and the experimental results indicated the improvement of NM2H, while the overhead introduced was acceptable.