Boosting Atmospheric Dust Forecast with PyCOMPSs

Task-based programming is becoming a tool of large interest for boosting High-Performance Computing (HPC) and Big Data applications. In particular, COMP Superscalar (COMPSs), is showing to be an effective task-based programming model for distributed computing of Big Data applications within HPC environments. Applications like NMMB-MONARCH, which is a dust forecast application composed by a set of steps (being some of them binaries with or without MPI), are perfect candidates for PyCOMPSs, the Python binding of COMPSs. This paper describes the success story of the adaptation of the NMMB-MONARCH online multi-scale atmospheric dust model to PyCOMPSs in order to exploit its inherent parallelism with the minimal developer effort. The paper also includes an evaluation of this implementation in the Nord3 supercomputer, a scalability analysis and an in-depth behaviour study. The main results presented in this paper are: (1) PyCOMPSs is able to extract the parallelism from the NMMB-MONARCH application; (2) it is able to improve the dust forecasting in terms of performance when compared with previous versions, and (3) PyCOMPSs is able to interact and share the resources with MPI applications when included in the workflow as tasks. Finally, we present the keys for exporting the knowledge of this experience to other applications in order to benefit from using PyCOMPSs.