Teaching MPI from Mental Models

Abstract

The Message Passing Interface (MPI) is the de facto standard for programming large scale parallelism, with up to millions of individual processes. Its dominant paradigm of Single Program Multiple Data (SPMD) programming is different from threaded and multicore parallelism, to an extent that students have a hard time switching models. In contrast to threaded programming, which allows for a view of the execution with central control and a central repository of data, SPMD programming has a symmetric model where all processes are active all the time, and none is priviliged in any sense, and where data is distributed.This model is counterintuitive to the novice parallel programmer, so care needs to be taken how to instill the proper ‘mental model'.We identify problems with the currently common way of teaching MPI, and propose a way that is geared to explicit reinforcing the symmetric model. Additionally, we teach starting from realistic scenarios, rather than writing artificial code just to exercise a newly-learned routine.This motivation implies that we reverse the commonly used order of presenting MPI routines, starting with collectives, and later introducing point-to-point routines only as support for certain symmetric operations, avoiding the process-to-process model.