Engineering resource management middleware for optimizing the performance of clouds processing mapreduce jobs with deadlines

This paper focuses on devising efficient resource management techniques used by the resource management middleware in clouds that handle MapReduce jobs with end-to-end service level agreements (SLAs) comprising an earliest start time, execution time, and a deadline. This research and development work, performed in collaboration with our industrial partner, presents the formulation of the matchmaking and scheduling problem for MapReduce jobs as an optimization problem using: Mixed Integer Linear Programming (MILP) and Constraint Programming (CP) techniques. In addition to the formulations devised, our experience in implementing the MILP and CP models using various open source as well as commercial software packages is described. Furthermore, a performance evaluation of the different approaches used to implement the formulations is conducted using a variety of different workloads.