Accelerating SWAT Simulations Using An In-Memory NoSQL Database

IF 6 1区 环境科学与生态学 Q1 ENVIRONMENTAL SCIENCES
D. Zhang, Q. Lin, H. Yao, Y. R. He, J. Deng, X. X. Zhang, Bellwood Road Dorset P A E Ontario Canada Parks
{"title":"Accelerating SWAT Simulations Using An In-Memory NoSQL Database","authors":"D. Zhang, Q. Lin, H. Yao, Y. R. He, J. Deng, X. X. Zhang, Bellwood Road Dorset P A E Ontario Canada Parks","doi":"10.3808/jei.201900425","DOIUrl":null,"url":null,"abstract":"Due to its versatility, the Soil and Water Assessment Tool (SWAT) has been widely applied to investigate the effects of management activities and climate change on water availability and quality. However, the use of high spatial resolution data and the advantages of SWAT itself have significantly increased the input/output (I/O) demand and thus the runtime of modeling routines that require a large number of iterative simulations. In this study, we proposed a generic scheme to reduce the SWAT runtime by caching the model inputs using the in-memory NoSQL database Redis. Then the SWAT source codes (rev 488) was modified according to this proposed scheme to develop the MA-SWAT (memory accelerated SWAT) model by incorporating a new subroutine known as Fortran_calls_c to retrieve the cached inputs. We then evaluated MA-SWAT with four synthetic hydrological models and five different parallel schemes in a quad-core commodity laptop. The test results showed that when applied with a parallel simulation program, MA-SWAT could achieve a speedup by a factor of 8.4 ~ 10.9 depending on model complexity. Compared with the original SWAT, MA-SWAT significantly improved the computation speed, indicating that the proposed scheme is a desirable method for solving high computational demand problems such as calibration, sensitivity and uncertainty analysis. Moreover, the proposed concept of linking the SWAT model with Redis via the minimalistic C client driver of Redis is a generic method, and it is possible to apply this method to other Fortran-implemented environmental model to alleviate I/O demands.","PeriodicalId":54840,"journal":{"name":"Journal of Environmental Informatics","volume":"6 1","pages":""},"PeriodicalIF":6.0000,"publicationDate":"2019-12-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"5","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Environmental Informatics","FirstCategoryId":"93","ListUrlMain":"https://doi.org/10.3808/jei.201900425","RegionNum":1,"RegionCategory":"环境科学与生态学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"ENVIRONMENTAL SCIENCES","Score":null,"Total":0}
引用次数: 5

Abstract

Due to its versatility, the Soil and Water Assessment Tool (SWAT) has been widely applied to investigate the effects of management activities and climate change on water availability and quality. However, the use of high spatial resolution data and the advantages of SWAT itself have significantly increased the input/output (I/O) demand and thus the runtime of modeling routines that require a large number of iterative simulations. In this study, we proposed a generic scheme to reduce the SWAT runtime by caching the model inputs using the in-memory NoSQL database Redis. Then the SWAT source codes (rev 488) was modified according to this proposed scheme to develop the MA-SWAT (memory accelerated SWAT) model by incorporating a new subroutine known as Fortran_calls_c to retrieve the cached inputs. We then evaluated MA-SWAT with four synthetic hydrological models and five different parallel schemes in a quad-core commodity laptop. The test results showed that when applied with a parallel simulation program, MA-SWAT could achieve a speedup by a factor of 8.4 ~ 10.9 depending on model complexity. Compared with the original SWAT, MA-SWAT significantly improved the computation speed, indicating that the proposed scheme is a desirable method for solving high computational demand problems such as calibration, sensitivity and uncertainty analysis. Moreover, the proposed concept of linking the SWAT model with Redis via the minimalistic C client driver of Redis is a generic method, and it is possible to apply this method to other Fortran-implemented environmental model to alleviate I/O demands.
使用内存NoSQL数据库加速SWAT模拟
由于其通用性,水土评价工具(SWAT)已被广泛应用于研究管理活动和气候变化对水的可利用性和质量的影响。然而,高空间分辨率数据的使用和SWAT本身的优势大大增加了输入/输出(I/O)需求,从而增加了需要大量迭代模拟的建模例程的运行时间。在本研究中,我们提出了一种通用方案,通过使用内存NoSQL数据库Redis缓存模型输入来减少SWAT运行时。然后,根据该方案对SWAT源代码(rev 488)进行修改,通过添加新的子程序Fortran_calls_c来检索缓存的输入,从而开发出内存加速SWAT (memory accelerated SWAT)模型。然后,我们在一台四核商用笔记本电脑上用四种合成水文模型和五种不同的并行方案评估了MA-SWAT。测试结果表明,当与并行仿真程序结合使用时,根据模型复杂度的不同,MA-SWAT可以实现8.4 ~ 10.9倍的加速。与原有的SWAT算法相比,MA-SWAT算法显著提高了计算速度,表明该方案是解决校准、灵敏度和不确定度分析等高计算需求问题的理想方法。此外,通过Redis的极简C客户端驱动程序将SWAT模型与Redis连接的概念是一种通用方法,并且可以将这种方法应用于其他fortran实现的环境模型以减轻I/O需求。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
Journal of Environmental Informatics
Journal of Environmental Informatics ENVIRONMENTAL SCIENCES-
CiteScore
12.40
自引率
2.90%
发文量
7
审稿时长
24 months
期刊介绍: Journal of Environmental Informatics (JEI) is an international, peer-reviewed, and interdisciplinary publication designed to foster research innovation and discovery on basic science and information technology for addressing various environmental problems. The journal aims to motivate and enhance the integration of science and technology to help develop sustainable solutions that are consensus-oriented, risk-informed, scientifically-based and cost-effective. JEI serves researchers, educators and practitioners who are interested in theoretical and/or applied aspects of environmental science, regardless of disciplinary boundaries. The topics addressed by the journal include: - Planning of energy, environmental and ecological management systems - Simulation, optimization and Environmental decision support - Environmental geomatics - GIS, RS and other spatial information technologies - Informatics for environmental chemistry and biochemistry - Environmental applications of functional materials - Environmental phenomena at atomic, molecular and macromolecular scales - Modeling of chemical, biological and environmental processes - Modeling of biotechnological systems for enhanced pollution mitigation - Computer graphics and visualization for environmental decision support - Artificial intelligence and expert systems for environmental applications - Environmental statistics and risk analysis - Climate modeling, downscaling, impact assessment, and adaptation planning - Other areas of environmental systems science and information technology.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信