Mixed Data-Parallel Scheduling for Distributed Continuous Integration

Abstract

In this paper, we consider the problem of scheduling a special kind of mixed data-parallel applications arising in the context of Continuous Integration. Continuous integration (CI) is a software engineering technique, which consists in re-building and testing interdependent software components as soon as developers modify them. The CI tool is able to provide quick feedback to the developers, which allows them to fix the bug soon after it has been introduced. The CI process can be described as a DAG where nodes represent package build tasks, and edges represent dependencies among these packages, build tasks themselves can in turn be run in parallel. Thus, CI can be viewed as a mixed data-parallel application. A crucial point for a successful CI process is its ability to provide quick feedback. Thus, make span minimization is the main goal. Our contribution is twofold. First we provide and analyze a large dataset corresponding to a build DAG. Second, we compare the performance of several scheduling heuristics on this dataset.