Big data analysis on yelp user-generated reviews

Abstract

The goal of this project is to demostrate the use of PySpark and Spark SQL to query and analyze the Yelp Open Dataset. Specifically, the aim is to analyze the Yelp Reviews dataset, which consists of 8.6 million user-generated reviews of businesses on Yelp. we also perform JOIN operations with the Yelp Business and Yelp User datasets to describe relations between review ratings and characteristics of the business, such as geographic location. To perform some of these queries, we demonstrate the use of user-defined functions (UDFs) in Spark SQL queries. Lastly, we briefly examine how partitioning of the underlying data abstraction changes computational speed.