Big Data Processing using Machine Learning algorithms: MLlib and Mahout Use Case

Abstract

Machine learning is a field within artificial intelligence that allows machines to learn on their own from existing information to make predictions or/and decisions. There are three main categories of machine learning techniques: Collaborative filtering (for making recommendations), Clustering (for discovering structure in collections of data) and Classification (form of supervised learning). Machine learning helps users to make better decisions, Machine learning algorithms create patterns based on previous information and use them to design predictive models, then, use this models to obtain predictions about future data. A huge amount of data from several sources need methods and techniques to be processed correctly, in order to exploit this data efficiently, machine learning is a great technology for exploiting the needs in big data analysis. This paper describes the implementation of Apache Spark MLlib and Apache Mahout in order to process Big Data using Machine Learning algorithms. Furthermore, we conduct experimental simulations to show the difference between this two Machine Learning frameworks. Subsequently, we discuss the most striking observations that emerge from the comparison of these technologies through several experimental studies.