Online data analysis and reduction: An important Co-design motif for extreme-scale computers

IF 3.5 3区计算机科学 Q2 COMPUTER SCIENCE, HARDWARE & ARCHITECTURE

International Journal of High Performance Computing Applications Pub Date : 2021-06-12 DOI:10.1177/10943420211023549

Ian Foster, M. Ainsworth, J. Bessac, F. Cappello, J. Choi, S. Di, Z. Di, A. M. Gok, Hanqi Guo, K. Huck, Christopher Kelly, S. Klasky, K. Kleese van Dam, Xin Liang, Kshitij Mehta, M. Parashar, T. Peterka, Line C. Pouchard, Tong Shu, O. Tugluk, H. V. van Dam, Lipeng Wan, Matthew Wolf, J. Wozniak, Wei Xu, I. Yakushin, Shinjae Yoo, T. Munson

{"title":"Online data analysis and reduction: An important Co-design motif for extreme-scale computers","authors":"Ian Foster, M. Ainsworth, J. Bessac, F. Cappello, J. Choi, S. Di, Z. Di, A. M. Gok, Hanqi Guo, K. Huck, Christopher Kelly, S. Klasky, K. Kleese van Dam, Xin Liang, Kshitij Mehta, M. Parashar, T. Peterka, Line C. Pouchard, Tong Shu, O. Tugluk, H. V. van Dam, Lipeng Wan, Matthew Wolf, J. Wozniak, Wei Xu, I. Yakushin, Shinjae Yoo, T. Munson","doi":"10.1177/10943420211023549","DOIUrl":null,"url":null,"abstract":"A growing disparity between supercomputer computation speeds and I/O rates means that it is rapidly becoming infeasible to analyze supercomputer application output only after that output has been written to a file system. Instead, data-generating applications must run concurrently with data reduction and/or analysis operations, with which they exchange information via high-speed methods such as interprocess communications. The resulting parallel computing motif, online data analysis and reduction (ODAR), has important implications for both application and HPC systems design. Here we introduce the ODAR motif and its co-design concerns, describe a co-design process for identifying and addressing those concerns, present tools that assist in the co-design process, and present case studies to illustrate the use of the process and tools in practical settings.","PeriodicalId":54957,"journal":{"name":"International Journal of High Performance Computing Applications","volume":"35 1","pages":"617 - 635"},"PeriodicalIF":3.5000,"publicationDate":"2021-06-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1177/10943420211023549","citationCount":"14","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"International Journal of High Performance Computing Applications","FirstCategoryId":"94","ListUrlMain":"https://doi.org/10.1177/10943420211023549","RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"COMPUTER SCIENCE, HARDWARE & ARCHITECTURE","Score":null,"Total":0}

引用次数: 14

Abstract

A growing disparity between supercomputer computation speeds and I/O rates means that it is rapidly becoming infeasible to analyze supercomputer application output only after that output has been written to a file system. Instead, data-generating applications must run concurrently with data reduction and/or analysis operations, with which they exchange information via high-speed methods such as interprocess communications. The resulting parallel computing motif, online data analysis and reduction (ODAR), has important implications for both application and HPC systems design. Here we introduce the ODAR motif and its co-design concerns, describe a co-design process for identifying and addressing those concerns, present tools that assist in the co-design process, and present case studies to illustrate the use of the process and tools in practical settings.

查看原文本刊更多论文

在线数据分析和约简：超大规模计算机的重要协同设计主题

超级计算机计算速度和I/O速率之间的差距越来越大，这意味着只有在将超级计算机应用程序输出写入文件系统后才能分析该输出，这很快变得不可行。相反，数据生成应用程序必须与数据缩减和/或分析操作同时运行，它们通过进程间通信等高速方法与这些操作交换信息。由此产生的并行计算主题，在线数据分析和约简（ODAR），对应用程序和HPC系统设计都有重要意义。在这里，我们介绍ODAR主题及其共同设计关注点，描述用于识别和解决这些关注点的共同设计过程，介绍有助于共同设计过程的工具，并介绍案例研究，以说明该过程和工具在实际环境中的使用。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

International Journal of High Performance Computing Applications 工程技术-计算机：跨学科应用

CiteScore

6.10

自引率

6.50%

发文量

审稿时长

>12 weeks

期刊介绍： With ever increasing pressure for health services in all countries to meet rising demands, improve their quality and efficiency, and to be more accountable; the need for rigorous research and policy analysis has never been greater. The Journal of Health Services Research & Policy presents the latest scientific research, insightful overviews and reflections on underlying issues, and innovative, thought provoking contributions from leading academics and policy-makers. It provides ideas and hope for solving dilemmas that confront all countries.