Hadoop in OpenStack: Data-location-aware cluster provisioning

Abstract

Nowadays, cloud based analytics platforms are replacing traditional physical clusters due to the high efficiency it provides. Such cloud platforms runs Hadoop on virtual clusters with remotely attached storage. In cloud architecture with multiple geographically separated regions, virtual machines (VMs) belonging to a virtual cluster are placed randomly. In order to run MapReduce jobs, data have to be moved to the regions where the VMs reside to achieve data locality. In this paper, we propose a data-location aware virtual cluster provisioning strategy to identify the data location and provision the cluster near to the storage. The use of bio-inspired optimization algorithms are considered for optimizing the placements of VMs. Data location aware cluster provisioning reduces the network distance between storage and the virtual cluster, resulting in faster job completion times.