Numba-MPI v1.0: Enabling MPI communication within Numba/LLVM JIT-compiled Python code

IF 2.4 4区计算机科学 Q2 COMPUTER SCIENCE, SOFTWARE ENGINEERING

SoftwareX Pub Date : 2024-10-04 DOI:10.1016/j.softx.2024.101897

Kacper Derlatka , Maciej Manna , Oleksii Bulenok , David Zwicker , Sylwester Arabas

{"title":"Numba-MPI v1.0: Enabling MPI communication within Numba/LLVM JIT-compiled Python code","authors":"Kacper Derlatka , Maciej Manna , Oleksii Bulenok , David Zwicker , Sylwester Arabas","doi":"10.1016/j.softx.2024.101897","DOIUrl":null,"url":null,"abstract":"<div><div>The numba-mpi package offers access to the Message Passing Interface (MPI) routines from Python code that uses the Numba just-in-time (JIT) compiler. As a result, high-performance and multi-threaded Python code may utilize MPI communication facilities without leaving the JIT-compiled code blocks, which is not possible with the mpi4py package, a higher-level Python interface to MPI. For debugging or code-coverage analysis purposes, numba-mpi retains full functionality of the code even if the JIT compilation is disabled. The numba-mpi API constitutes a thin wrapper around the C API of MPI and is built around Numpy arrays including handling of non-contiguous views over array slices. Project development is hosted at GitHub leveraging the mpi4py/setup-mpi workflow enabling continuous integration tests on Linux (MPICH, OpenMPI & Intel MPI), macOS (MPICH & OpenMPI) and Windows (MS MPI). The paper covers an overview of the package features, architecture and performance. As of v1.0, the following MPI routines are exposed and covered by unit tests: size/rank, [i]send/[i]recv, wait[all|any], test[all|any], allreduce, bcast, barrier, scatter/[all]gather & wtime. The package is implemented in pure Python and depends on numpy, numba and mpi4py (the latter used at initialization and as a source of utility routines only). The performance advantage of using numba-mpi compared to mpi4py is depicted with a simple example, with entirety of the code included in listings discussed in the text. Application of numba-mpi for handling domain decomposition in numerical solvers for partial differential equations is presented using two external packages that depend on numba-mpi: py-pde and PyMPDATA-MPI.</div></div>","PeriodicalId":21905,"journal":{"name":"SoftwareX","volume":"28 ","pages":"Article 101897"},"PeriodicalIF":2.4000,"publicationDate":"2024-10-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"SoftwareX","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S235271102400267X","RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"COMPUTER SCIENCE, SOFTWARE ENGINEERING","Score":null,"Total":0}

引用次数: 0

Abstract

The numba-mpi package offers access to the Message Passing Interface (MPI) routines from Python code that uses the Numba just-in-time (JIT) compiler. As a result, high-performance and multi-threaded Python code may utilize MPI communication facilities without leaving the JIT-compiled code blocks, which is not possible with the mpi4py package, a higher-level Python interface to MPI. For debugging or code-coverage analysis purposes, numba-mpi retains full functionality of the code even if the JIT compilation is disabled. The numba-mpi API constitutes a thin wrapper around the C API of MPI and is built around Numpy arrays including handling of non-contiguous views over array slices. Project development is hosted at GitHub leveraging the mpi4py/setup-mpi workflow enabling continuous integration tests on Linux (MPICH, OpenMPI & Intel MPI), macOS (MPICH & OpenMPI) and Windows (MS MPI). The paper covers an overview of the package features, architecture and performance. As of v1.0, the following MPI routines are exposed and covered by unit tests: size/rank, [i]send/[i]recv, wait[all|any], test[all|any], allreduce, bcast, barrier, scatter/[all]gather & wtime. The package is implemented in pure Python and depends on numpy, numba and mpi4py (the latter used at initialization and as a source of utility routines only). The performance advantage of using numba-mpi compared to mpi4py is depicted with a simple example, with entirety of the code included in listings discussed in the text. Application of numba-mpi for handling domain decomposition in numerical solvers for partial differential equations is presented using two external packages that depend on numba-mpi: py-pde and PyMPDATA-MPI.

查看原文本刊更多论文

Numba-MPI v1.0：在Numba/LLVM JIT编译的Python代码中启用MPI通信

numba-mpi 软件包提供了从使用 Numba 即时（JIT）编译器的 Python 代码访问消息传递接口（MPI）例程的功能。因此，高性能和多线程 Python 代码可以在不离开 JIT 编译代码块的情况下使用 MPI 通信设施，而 mpi4py 软件包（MPI 的高级 Python 接口）则无法做到这一点。出于调试或代码覆盖分析的目的，即使禁用了 JIT 编译，numba-mpi 也能保留代码的全部功能。numba-mpi API 是对 MPI C API 的精简封装，围绕 Numpy 数组构建，包括处理数组切片上的非连续视图。项目开发托管在 GitHub 上，利用 mpi4py/setup-mpi 工作流在 Linux（MPICH、OpenMPI & Intel MPI）、macOS（MPICH & OpenMPI）和 Windows（MS MPI）上进行持续集成测试。本文概述了软件包的功能、架构和性能。截至 v1.0，以下 MPI 例程已公开并通过单元测试：size/rank、[i]send/[i]recv、wait[all|any]、test[all|any]、allreduce、bcast、barrier、scatter/[all]gather & wtime。该软件包以纯 Python 实现，依赖于 numpy、numba 和 mpi4py（后者仅在初始化时使用，并作为实用例程的源代码）。与 mpi4py 相比，使用 numba-mpi 在性能上的优势将通过一个简单的示例来说明，整个代码包含在文中讨论的列表中。我们使用两个依赖于 numba-mpi 的外部软件包：py-pde 和 PyMPDATA-MPI，介绍了 numba-mpi 在偏微分方程数值求解器中处理域分解的应用。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

SoftwareX COMPUTER SCIENCE, SOFTWARE ENGINEERING-

CiteScore

5.50

自引率

2.90%

发文量

184

审稿时长

9 weeks

期刊介绍： SoftwareX aims to acknowledge the impact of software on today''s research practice, and on new scientific discoveries in almost all research domains. SoftwareX also aims to stress the importance of the software developers who are, in part, responsible for this impact. To this end, SoftwareX aims to support publication of research software in such a way that: The software is given a stamp of scientific relevance, and provided with a peer-reviewed recognition of scientific impact; The software developers are given the credits they deserve; The software is citable, allowing traditional metrics of scientific excellence to apply; The academic career paths of software developers are supported rather than hindered; The software is publicly available for inspection, validation, and re-use. Above all, SoftwareX aims to inform researchers about software applications, tools and libraries with a (proven) potential to impact the process of scientific discovery in various domains. The journal is multidisciplinary and accepts submissions from within and across subject domains such as those represented within the broad thematic areas below: Mathematical and Physical Sciences; Environmental Sciences; Medical and Biological Sciences; Humanities, Arts and Social Sciences. Originating from these broad thematic areas, the journal also welcomes submissions of software that works in cross cutting thematic areas, such as citizen science, cybersecurity, digital economy, energy, global resource stewardship, health and wellbeing, etcetera. SoftwareX specifically aims to accept submissions representing domain-independent software that may impact more than one research domain.