Deploying and researching Hadoop in virtual machines

Abstract

Hadoop's emerging and the maturity of virtualization make it feasible to combine them together to process immense data set. To do research on Hadoop in virtual environment, an experimental environment is needed. This paper firstly introduces some technologies used such as CloudStack, MapReduce and Hadoop. Based on that, a method to deploy CloudStack is given. Then we discuss how to deploy Hadoop in virtual machines which can be obtained from CloudStack by some means, then an algorithm to solve the problem that all the virtual machines which are created by CloudStack using same template have a same hostname. After that we run some Hadoop programs under the virtual cluster, which shows that it is feasible to deploying Hadoop in this way. Then some methods to optimize Hadoop in virtual machines are discussed. From this paper, readers can follow it to set up their own Hadoop experimental environment and capture the current status and trend of optimizing Hadoop in virtual environment.