A Cloud-Based Batch Processing System for Loosely-Coupled Applications

With the increased amount of data available for processing, and the increased need of processing this data, loosely-coupled batch applications have become very popular. Many batch applications require a high level of processing capacity, which leads us to the need of high performance computing infrastructures. This approach has been used for a long time, mainly for scientific purposes, and focused on the conventional environments for HPC, namely local clusters and supercomputers. The high-speed networks present in these systems are paramount for the execution of tightly-coupled scientific applications, but are wasted when executing loosely-coupled applications. Cloud infrastructures, on the other hand, provide a more appropriate infrastructure to support such loosely-coupled applications. Unfortunately, the user experience in cloud systems is completely different from that of conventional batch systems, mainly because the infrastructure needs to be deployed and subsequently released, to achieve the desired gains. In this paper we propose the architecture of a batch processing system that takes advantage of common features of cloud infrastructures to minimize cost and waiting time, while providing a user experience that is similar to conventional HPC systems.