A Front-end, Hadoop-based Data Management Service for Efficient Federated Clouds

Abstract

In the recent years, cloud computing has emerged as the new IT paradigm that promises elastic resources on a pay-per-use basis. The challenges of cloud computing are focused around massive data storage and efficient large scale distributed computation. Hadoop, a community driven Apache project has provided an efficient and cost effective platform for large scale computation using the map-reduce methodology, pioneered by Google. In this paper, the design of a Hadoop-based data management system as the front-end service for Cloud data management is investigated. This framework is enriched with Restful APIs in front of Hadoop and a series of components that aim to extend Hadoop's functionality beyond its well known back-end, heavy data processing scope. These components are used to enrich security, logging and data analysis features and also data access compatibility between different but interconnected Cloud providers (federated Clouds). Hadoop capabilities are also extended in a quest for intelligent decision making regarding the choice of the fittest services for federation in a federated cloud scenario, in addition to legally compliant behaviour regarding the geographical location of data storage.