Optimizing Hadoop parameter for speedup using Q-Learning Reinforcement Learning

Abstract

Hadoop is the most popular open source big data processing platform which is being used in many big data analytics applications. The performance of Hadoop can be fine-tuned for application performance requirements by adjusting the value of the some of the configuration parameters. Various methods have been proposed in literature for fine tuning the configuration parameters of Hadoop. The relation between the Hadoop performance tuning parameters and speed up is dependent on the nature of the applications and environment dynamics. Tuning the parameters without consideration of these dynamics results in sub optimal configurations and lower performance.. Adaptive reinforcement learning using Q-Learning is proposed in this work to fine tune the configuration parameters with the objective of reducing the error between desired and achieved service level agreement (SLA).