Multiple big data processing platforms

Abstract

The integration of Hive, Impala and Spark SQL platforms has achieved to perform rapid data retrieval using SQL query in big data environment. This paper is to design the optimized platform selection for highly improving the response of data retrieval. It can automatically choose the best-perform platform to best perform SQL commands. In addition, the distributed memory storage systems using Memcached and the distributed file system Hadoop HDFS have implemented the caching so that the fastest data retrieval has done once the repeated SQL command has applied.