Inter-Job Scheduling of High-Throughput Material Screening Applications

Abstract

Material screening entails a large number of electronic structure simulations. Traditionally, these simulation runs are treated separately as solving independent Kohn-Sham (KS) equations. In this paper, we formulate material screening as an inter-job scheduling problem for solving a system of KS equations, and in doing so allowing one to explore different scheduling methods that use the results of some equations to expedite the solution of others. We propose the concept of sharing iterative simulation and employ several optimization methods to initialize a simulation run using the distribution of particles from similar jobs as the initial condition. More specifically, we propose two similarity metrics, one qualitative and the other quantitative, to predict the simulation runtime of a material screen job based on its similarity to other jobs. Accordingly, we present two inter-job scheduling algorithms that make use the qualitative and quantitative similarity information. We conducted extensive experiments on the Sunway TaihuLight supercomputer for a practical material screening problem to evaluate the performance of the two scheduling algorithms using the proposed similarity metrics. We show that the total time required to run the large number of material screening jobs can be significantly reduced, and the algorithms are robust even with moderate inaccurate prediction on the simulation runtime. The quantitative algorithm achieves better results than the qualitative algorithm using more accurate prediction and thus achieving more significant runtime reduction.