CnC-Hadoop: a graphical coordination language for distributed multiscale parallelism

Abstract

The information-technology platform is being radically transformed with the widespread adoption of the cloud computing model supported by data centers containing large numbers of multicore servers. While cloud computing platforms can potentially enable a rich variety of distributed applications, the need to exploit multiscale parallelism at the inter-node and intra-node level poses significantly new challenges for software. Recent advances in the Google MapReduce and Hadoop frameworks have led to simplified programming models for a restricted class of distributed batch-processing applications. However, these frameworks do not support richer distributed application structures beyond map-reduce, and do not offer any solutions for exploiting shared-memory multicore parallelism at the intra-node level.