MCMC generation of cost matrices for scheduling performance evaluation

In high performance computing, scheduling and allocating tasks to machines has long been a critical challenge, especially when dealing with heterogeneous execution costs. To design efficient algorithms and then assess their performance, many approaches have been proposed, among which simulations, which can be performed on a large variety of environments and application models. However, this technique is known to be sensitive to bias when it relies on random instances with an uncontrolled distribution. In this article, instead of designing a new optimization method, we focus on generating cost matrices to improve the empirical evaluation methodology. In particular, we use methods from the literature to provide formal guarantee on how costs matrices are distributed: we ensure a uniform distribution among the cost matrices with given task and machine heterogeneities. Although the use of randomly generated matrices has often been criticized, this new generation procedure is the first that is proven to prevent biased generation by ensuring a uniform generation with given properties. This method is relevant to assess the performance of scheduling heuristics, in particular when characterizing for which parameter values a given approach performs better than others. When applied to a makespan minimization problem, the methodology reveals when each of three efficient heuristics performs better depending on the instance heterogeneity.