WRF 中的 I/O：现代并行 I/O 技术案例研究

Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis Pub Date : 2023-11-11 DOI:10.1145/3581784.3613216

Zanhua Huang, Kai-yuan Hou, Ankit Agrawal, Alok N. Choudhary, Robert Ross, W. Liao

{"title":"WRF 中的 I/O：现代并行 I/O 技术案例研究","authors":"Zanhua Huang, Kai-yuan Hou, Ankit Agrawal, Alok N. Choudhary, Robert Ross, W. Liao","doi":"10.1145/3581784.3613216","DOIUrl":null,"url":null,"abstract":"Large-scale parallel applications can face significant I/O performance bottlenecks, making efficient I/O crucial. This work presents a comparative study of several parallel I/O implementations in the Weather Research and Forecasting model, including PnetCDF blocking and non-blocking I/O options, netCDF4, HDF5 Log VOL, and ADIOS. For I/O methods creating files in a canonical data layout, PnetCDF's non-blocking option offers up to 2x improvement over its blocking option and up to 4.5x over HDF5 via netCDF4, demonstrating the effectiveness of the write request aggregation technique. The HDF5 Log VOL outperforms ADIOS with a 4x improvement in write performance when creating files in the log layout, although both require non-negligible time to convert the file back to canonical order for post-run analysis. From these results we extract some observations that can guide I/O strategies for modern parallel codes.","PeriodicalId":124077,"journal":{"name":"Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis","volume":"10 5","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2023-11-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"I/O in WRF: A Case Study in Modern Parallel I/O Techniques\",\"authors\":\"Zanhua Huang, Kai-yuan Hou, Ankit Agrawal, Alok N. Choudhary, Robert Ross, W. Liao\",\"doi\":\"10.1145/3581784.3613216\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Large-scale parallel applications can face significant I/O performance bottlenecks, making efficient I/O crucial. This work presents a comparative study of several parallel I/O implementations in the Weather Research and Forecasting model, including PnetCDF blocking and non-blocking I/O options, netCDF4, HDF5 Log VOL, and ADIOS. For I/O methods creating files in a canonical data layout, PnetCDF's non-blocking option offers up to 2x improvement over its blocking option and up to 4.5x over HDF5 via netCDF4, demonstrating the effectiveness of the write request aggregation technique. The HDF5 Log VOL outperforms ADIOS with a 4x improvement in write performance when creating files in the log layout, although both require non-negligible time to convert the file back to canonical order for post-run analysis. From these results we extract some observations that can guide I/O strategies for modern parallel codes.\",\"PeriodicalId\":124077,\"journal\":{\"name\":\"Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis\",\"volume\":\"10 5\",\"pages\":\"\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2023-11-11\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1145/3581784.3613216\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1145/3581784.3613216","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}

引用次数: 0

摘要

大规模并行应用可能面临严重的 I/O 性能瓶颈，因此高效 I/O 至关重要。这项工作对天气研究和预测模型中的几种并行 I/O 实现进行了比较研究，包括 PnetCDF 阻塞和非阻塞 I/O 选项、netCDF4、HDF5 Log VOL 和 ADIOS。对于以规范数据布局创建文件的 I/O 方法，PnetCDF 的非阻塞选项比其阻塞选项提高了 2 倍，通过 netCDF4 比 HDF5 提高了 4.5 倍，这证明了写请求聚合技术的有效性。在日志布局中创建文件时，HDF5 Log VOL 的写入性能比 ADIOS 高出 4 倍，不过两者都需要花费不可忽略的时间将文件转换回规范顺序，以便进行运行后分析。从这些结果中，我们得出了一些可以指导现代并行代码 I/O 策略的结论。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

查看原文本刊更多论文

I/O in WRF: A Case Study in Modern Parallel I/O Techniques

Large-scale parallel applications can face significant I/O performance bottlenecks, making efficient I/O crucial. This work presents a comparative study of several parallel I/O implementations in the Weather Research and Forecasting model, including PnetCDF blocking and non-blocking I/O options, netCDF4, HDF5 Log VOL, and ADIOS. For I/O methods creating files in a canonical data layout, PnetCDF's non-blocking option offers up to 2x improvement over its blocking option and up to 4.5x over HDF5 via netCDF4, demonstrating the effectiveness of the write request aggregation technique. The HDF5 Log VOL outperforms ADIOS with a 4x improvement in write performance when creating files in the log layout, although both require non-negligible time to convert the file back to canonical order for post-run analysis. From these results we extract some observations that can guide I/O strategies for modern parallel codes.

求助全文

通过发布文献求助，成功后即可免费获取论文全文。去求助

来源期刊

Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

自引率

0.00%

发文量