2021 3rd International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC)最新文献

Feasibility of Running Singularity Containers with Hybrid MPI on NASA High-End Computing Resources 混合MPI在NASA高端计算资源上运行奇点容器的可行性

2021 3rd International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC) Pub Date : 2021-11-01 DOI: 10.1109/CANOPIEHPC54579.2021.00007

Y. Chang, S. Heistand, R. Hood, Henry Jin

{"title":"Feasibility of Running Singularity Containers with Hybrid MPI on NASA High-End Computing Resources","authors":"Y. Chang, S. Heistand, R. Hood, Henry Jin","doi":"10.1109/CANOPIEHPC54579.2021.00007","DOIUrl":"https://doi.org/10.1109/CANOPIEHPC54579.2021.00007","url":null,"abstract":"This work investigates the feasibility of a Singularity container-based solution to support a customizable computing environment for running users' MPI applications in “hybrid” MPI mode-where the MPI on the host machine works in tandem with MPI inside the container-on NASA's High-End Computing Capability (HECC) resources. Two types of real-world applications were tested: traditional High-Performance Computing (HPC) and Artificial Intelligence/Machine Learning (AI/ML). On the traditional HPC side, two JEDI containers built with Intel MPI for Earth science modeling were tested on both HECC in-house and HECC AWS Cloud CPU resources. On the AI/ML side, a NVIDIA TensorFlow container built with OpenMPI was tested with a Neural Collaborative Filtering recommender system and the ResNet-50 computer image system on the HECC in-house V100 GPUs. For each of these applications and resource environments, multiple hurdles were overcome after lengthy debugging efforts. Among them, the most significant ones were due to the conflicts between a host MPI and a container MPI and the complexity of the communication layers underneath. Although porting containers to run with a single node using just the container MPI is quite straightforward, our exercises demonstrate that running across multiple nodes in hybrid MPI mode requires knowledge of Singularity, MPI libraries, the operating system image, and the communication infrastructure such as the transport and network layers, which are traditionally handled by support staff of HPC centers and hardware or software vendors. In conclusion, porting and running Singularity containers on HECC resources or other data centers with similar environments is feasible but most users would need help to run them in hybrid MPI mode.","PeriodicalId":237957,"journal":{"name":"2021 3rd International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC)","volume":"72 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130870887","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 5

[Copyright notice] (版权)

2021 3rd International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC) Pub Date : 2021-11-01 DOI: 10.1109/canopiehpc54579.2021.00002

引用次数: 0

It's a Scheduling Affair: GROMACS in the Cloud with the KubeFlux Scheduler 这是一个调度事件:使用KubeFlux调度器在云中运行GROMACS

2021 3rd International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC) Pub Date : 2021-11-01 DOI: 10.1109/CANOPIEHPC54579.2021.00006

Claudia Misale, M. Drocco, Daniel J. Milroy, Carlos Eduardo Arango Gutierrez, Stephen Herbein, D. Ahn, Yoonho Park

引用次数: 8

RollingGantryCrane: Automation for unpacking containers into HPC environments 滚动式龙门起重机:用于在高性能计算环境中打开集装箱的自动化

2021 3rd International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC) Pub Date : 2021-11-01 DOI: 10.1109/CANOPIEHPC54579.2021.00008

Gregory J. Zynda, S. Gopaulakrishnan, John M. Fonner

引用次数: 3

Proceedings of CANOPIE-HPC 2021: 3rd International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC CANOPIE-HPC会议录2021:第三届HPC隔离环境容器和新编排范式国际研讨会

2021 3rd International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC) Pub Date : 2021-11-01 DOI: 10.1109/canopiehpc54579.2021.00001

引用次数: 0

Message from the Workshop Chairs 来自研讨会主席的信息

2021 3rd International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC) Pub Date : 2021-11-01 DOI: 10.1109/canopiehpc54579.2021.00004

Whpcsc

引用次数: 0