lpData: A Data Placement for High-Throughput and Low-Latency

Abstract

In the era of data explosion, many storage technologies have emerged for processing and analyzing the big data. Structured storage such as Parquet has highthroughput in sequential read while semi-structured storage such as HBase supports low-latency in random access. However, due to the gap between the two kinds of storage, neither of the storage is suitable to each other's application scenarios. Motivated by applications that need both access patterns, the work proposed a new data placement, that is, lpData. Inheriting efficient record-level index from Lucene and high-throughput file format from Parquet, lpData is able to speed up queries with predicates and guarantees the performance in sequential read at the same time. According to experimental results of this study, a) shows high performance in both sequential read and random access, b) compared to Parquet, lpData executes 42% faster on average in TPC-H selective queries, c) compared to Lucene, lpData outperforms 60% faster for low selective queries and by 36× for high selective queries on average.