Using compression tables to improve HiveQL Performance with Spark A Case study on NVMe Storage Devices

Abstract

Query language execution is widely used in big data. The SQL standard is the major query language. Big data has a lot of SQL-like tools, for example: Spark-SQL, Hive, Drill, and Presto. This paper focused on Hive with the Spark engine. To increase Hive’s query performance in a case study, NVMe Solid State Devices, we proposed the compressed Parquet file including SNAPPY, gzip, and Zstandard (zstd). Query workloads use TPC-H benchmark. Thus, this compression codec can reduce the main transaction table of TPC-H benchmark by 56%, and some queries have lower CPU usage than Text file. However, the Hive on Spark engine with our proposed compression codecs for Parquet files has lower CPU usage than Text file in some TPC-H queries. Thus, NVMe storage with the Parquet file compression codec is more efficient than text files for improving query performance on the Spark engine.