Demystifying Python Package Installation with conda-env-mod

Abstract

Novice users face significant challenges while installing and using Python packages in an HPC environment. Due to the inherent design of tools like Pip and Conda and how packages look for libraries, installing Python packages as a non-root user is complicated and often leads to broken packages with conflicting dependencies. With the growing popularity of Python in the HPC community, supporting users with their package installation needs is an evolving issue for the HPC center staff. In this paper, we present the design and implementation of conda-env-mod—a tool for simplifying the installation and use of Python packages in HPC clusters. conda-env-mod simplifies and streamlines the creation of virtual environments and provides users with environment modules for activating the environments. Users can install individual packages into isolated environments reducing chances of conflict (both current and future) and can activate multiple environments using modules as needed. After users load necessary modules, they can simply run pip and conda to install packages just like they would on their desktop. It also helps create Jupyter kernels and allows users to use external packages in a central JupyterHub installation with ease. conda-env-mod hides the complexity of configuring the package managers and setting up the users’ runtime environments and, thereby, reduces the barriers for novice Python users. Over the last three months (June-August, 2020), more than 160 users have used conda-env-mod to install and manage custom Python packages, while our deep learning package installations, facilitated by conda-env-mod, have been used by 60 plus users.