Stargate: a data source connector based on spark SQL

Abstract

Spark SQL has become a landing solution when a lot of enterprises in the face of massive data analysis and processing issues. To quickly and conveniently connect computing engines to data sources on different storage engines and help computing engine understanding and adapting the storage engine to improve the computational efficiency, we present a data source connector based on Spark SQL - Stargate, which provides a set of framework for different storage engine to connect to the Spark SQL computing engine. Experiments show that Stargate can perfectly match Spark SQL computing engine with multiple storage engines, and Stargate can help computing engines understand and adapt storage engines to improve the computational analysis efficiency.