Managed Network Services for Exascale Data Movement Across Large Global Scientific Collaborations

2022 4th Annual Workshop on Extreme-scale Experiment-in-the-Loop Computing (XLOOP) Pub Date : 2022-09-27 DOI:10.1109/XLOOP56614.2022.00008

F. Würthwein, J. Guiang, A. Arora, Diego Davila, John Graham, D. Mishin, Tom Hutton, I. Sfiligoi, Harvey Newman, J. Balcas, T. Lehman, Xi Yang, C. Guok

{"title":"Managed Network Services for Exascale Data Movement Across Large Global Scientific Collaborations","authors":"F. Würthwein, J. Guiang, A. Arora, Diego Davila, John Graham, D. Mishin, Tom Hutton, I. Sfiligoi, Harvey Newman, J. Balcas, T. Lehman, Xi Yang, C. Guok","doi":"10.1109/XLOOP56614.2022.00008","DOIUrl":null,"url":null,"abstract":"Unique scientific instruments designed and operated by large global collaborations are expected to produce Exabytescale data volumes per year by 2030. These collaborations depend on globally distributed storage and compute to turn raw data into science. While all of these infrastructures have batch scheduling capabilities to share compute, Research and Education networks lack those capabilities. There is thus uncontrolled competition for bandwidth between and within collaborations. As a result, data “hogs” disk space at processing facilities for much longer than it takes to process, leading to vastly over-provisioned storage infrastructures. Integrated co-scheduling of networks as part of high-level managed workflows might reduce these storage needs by more than an order of magnitude. This paper describes such a solution, demonstrates its functionality in the context of the Large Hadron Collider (LHC) at CERN, and presents the nextsteps towards its use in production.","PeriodicalId":401106,"journal":{"name":"2022 4th Annual Workshop on Extreme-scale Experiment-in-the-Loop Computing (XLOOP)","volume":"99 ","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2022-09-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"2022 4th Annual Workshop on Extreme-scale Experiment-in-the-Loop Computing (XLOOP)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/XLOOP56614.2022.00008","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}

引用次数: 0

Abstract

Unique scientific instruments designed and operated by large global collaborations are expected to produce Exabytescale data volumes per year by 2030. These collaborations depend on globally distributed storage and compute to turn raw data into science. While all of these infrastructures have batch scheduling capabilities to share compute, Research and Education networks lack those capabilities. There is thus uncontrolled competition for bandwidth between and within collaborations. As a result, data “hogs” disk space at processing facilities for much longer than it takes to process, leading to vastly over-provisioned storage infrastructures. Integrated co-scheduling of networks as part of high-level managed workflows might reduce these storage needs by more than an order of magnitude. This paper describes such a solution, demonstrates its functionality in the context of the Large Hadron Collider (LHC) at CERN, and presents the nextsteps towards its use in production.

查看原文本刊更多论文

跨大型全球科学合作的百亿亿级数据移动管理网络服务

由大型全球合作设计和操作的独特科学仪器预计到2030年每年将产生百亿亿次的数据量。这些合作依赖于全球分布式存储和计算，将原始数据转化为科学。虽然所有这些基础设施都具有批调度功能来共享计算，但研究和教育网络缺乏这些功能。因此，协作之间和协作内部存在着不受控制的带宽竞争。因此，数据在处理设施中“占用”磁盘空间的时间要比处理所需的时间长得多，从而导致存储基础设施供应严重过剩。作为高级管理工作流的一部分，集成的网络协同调度可能会将这些存储需求减少一个数量级以上。本文描述了这样一个解决方案，展示了它在欧洲核子研究中心大型强子对撞机(LHC)背景下的功能，并介绍了其在生产中使用的下一步。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

2022 4th Annual Workshop on Extreme-scale Experiment-in-the-Loop Computing (XLOOP)

自引率

0.00%

发文量