JETS: Language and System Support for Many-Parallel-Task Computing

Abstract

Many-task computing is a well-established paradigm for implementing loosely coupled applications on large-scale computing systems. However, few of the model's existing implementations provide efficient, low-latency support for the execution of tightly coupled applications as atomic tasks. Thus, a vast array of parallel applications cannot readily be used effectively within many-task workloads. In this work, we present JETS, a middleware component that provides high performance support for many-\emph{parallel}-task-computing (MPTC). JETS is based on a highly concurrent approach to parallel task dispatch and on new capabilities now available in the MPICH2 MPI implementation and the ZeptoOS Linux operating system. JETS represents an advancement over the few known examples of multi-level many-parallel-task scheduling systems by more efficiently scheduling many \emph{short-duration} parallel application invocations, by overcoming the challenges of coupling the user processes of each application invocation via the messaging fabric, and by concurrently managing many application executions in various stages. We report here on the JETS architecture and its performance on both synthetic benchmarks and the NAMD molecular dynamics application.