Boolean Logical Operator Driven Selective Data Filtering For Large Datasets

Abstract

Specific users of a system processing large data sets are often interested in only a small subset of the large volumes of available data. This paper presents research on a parallel processing based data filtering technique that filters out and stores only the subset of data that is of interest to a given user. A user’s preferences reflecting her/his interest are captured in a set of keywords or phrases which may be used in conjunction with Boolean operators. An Apache Spark based prototype is built and deployed on an Amazon EC2 cloud to demonstrate the viability of the approach and to analyze the performance of the proposed technique.