A surrogate model-based ESM parameter tuning scientific workflow management framework for HPC

IF 2.7 4区 地球科学 Q2 COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS
Liang Hu, Xianwei Wu, Xilong Che
{"title":"A surrogate model-based ESM parameter tuning scientific workflow management framework for HPC","authors":"Liang Hu, Xianwei Wu, Xilong Che","doi":"10.1007/s12145-024-01460-x","DOIUrl":null,"url":null,"abstract":"<p>In the present era, scientific computation is gradually becoming a primary research method, with an increasing number of researchers engaging in simulation studies on various high-performance computing platforms. Scientific workflows play a crucial role in organizing these complex research tasks effectively. However, poorly managed scientific workflows can lead to wastage of HPC computational resources and fail to alleviate the operational burden on researchers. The parameter optimization of Earth System Models (ESM) poses specific challenges due to its complexity, exacerbating these issues. To address these challenges, we propose a scientific workflow management framework for surrogate-based ESM parameter optimization. This framework consists of four layers: the resource layer, which gathers current resource information; the service layer, which provides various components to ensure the accurate execution of workflows; the management layer, which monitors the execution status of workflows; and the software environment interaction layer, which serves as the interface for data exchange between users and the framework. We monitored a team engaged in tuning CAM parameters before and after adopting the framework, and the results showed significant improvements in operation numbers, task execution time, and storage resource consumption after deploying the framework. This validates that our proposed scientific workflow management framework effectively addresses the challenges in user operations and resource management during surrogate-based ESM optimization processes, demonstrating the potential of our framework.</p>","PeriodicalId":49318,"journal":{"name":"Earth Science Informatics","volume":null,"pages":null},"PeriodicalIF":2.7000,"publicationDate":"2024-09-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Earth Science Informatics","FirstCategoryId":"89","ListUrlMain":"https://doi.org/10.1007/s12145-024-01460-x","RegionNum":4,"RegionCategory":"地球科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS","Score":null,"Total":0}
引用次数: 0

Abstract

In the present era, scientific computation is gradually becoming a primary research method, with an increasing number of researchers engaging in simulation studies on various high-performance computing platforms. Scientific workflows play a crucial role in organizing these complex research tasks effectively. However, poorly managed scientific workflows can lead to wastage of HPC computational resources and fail to alleviate the operational burden on researchers. The parameter optimization of Earth System Models (ESM) poses specific challenges due to its complexity, exacerbating these issues. To address these challenges, we propose a scientific workflow management framework for surrogate-based ESM parameter optimization. This framework consists of four layers: the resource layer, which gathers current resource information; the service layer, which provides various components to ensure the accurate execution of workflows; the management layer, which monitors the execution status of workflows; and the software environment interaction layer, which serves as the interface for data exchange between users and the framework. We monitored a team engaged in tuning CAM parameters before and after adopting the framework, and the results showed significant improvements in operation numbers, task execution time, and storage resource consumption after deploying the framework. This validates that our proposed scientific workflow management framework effectively addresses the challenges in user operations and resource management during surrogate-based ESM optimization processes, demonstrating the potential of our framework.

Abstract Image

基于代用模型的高性能计算ESM参数调整科学工作流管理框架
当今时代,科学计算逐渐成为一种主要的研究方法,越来越多的研究人员在各种高性能计算平台上从事模拟研究。科学工作流在有效组织这些复杂的研究任务方面发挥着至关重要的作用。然而,科学工作流管理不善会导致高性能计算资源的浪费,无法减轻研究人员的操作负担。地球系统模型(ESM)的参数优化因其复杂性而面临特殊挑战,加剧了这些问题。为了应对这些挑战,我们提出了一个基于代理的 ESM 参数优化科学工作流管理框架。该框架由四层组成:资源层,收集当前资源信息;服务层,提供各种组件以确保工作流的准确执行;管理层,监控工作流的执行状态;软件环境交互层,作为用户与框架之间的数据交换接口。在采用该框架前后,我们对参与调整 CAM 参数的团队进行了监测,结果显示,部署该框架后,操作数、任务执行时间和存储资源消耗都有了显著改善。这验证了我们提出的科学工作流管理框架有效地解决了基于代理的 ESM 优化过程中用户操作和资源管理方面的挑战,展示了我们框架的潜力。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
Earth Science Informatics
Earth Science Informatics COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS-GEOSCIENCES, MULTIDISCIPLINARY
CiteScore
4.60
自引率
3.60%
发文量
157
审稿时长
4.3 months
期刊介绍: The Earth Science Informatics [ESIN] journal aims at rapid publication of high-quality, current, cutting-edge, and provocative scientific work in the area of Earth Science Informatics as it relates to Earth systems science and space science. This includes articles on the application of formal and computational methods, computational Earth science, spatial and temporal analyses, and all aspects of computer applications to the acquisition, storage, processing, interchange, and visualization of data and information about the materials, properties, processes, features, and phenomena that occur at all scales and locations in the Earth system’s five components (atmosphere, hydrosphere, geosphere, biosphere, cryosphere) and in space (see "About this journal" for more detail). The quarterly journal publishes research, methodology, and software articles, as well as editorials, comments, and book and software reviews. Review articles of relevant findings, topics, and methodologies are also considered.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信