使用基于网格的点对点并行I/O进行负载平衡

Yijian Wang, D. Kaeli
{"title":"使用基于网格的点对点并行I/O进行负载平衡","authors":"Yijian Wang, D. Kaeli","doi":"10.1109/CLUSTR.2005.347040","DOIUrl":null,"url":null,"abstract":"In the area of grid computing, there is a growing need to process large amounts of data. To support this trend, we need to develop efficient parallel storage systems that can provide for high performance for data-intensive applications. In order to overcome I/O bottlenecks and to increase I/O parallelism, data streams need to be parallelized at both the application level and the storage device level. In this paper, we propose a novel peer-to-peer (P2P) storage architecture for MPI applications on grid systems. We first present an analytic model of our P2P storage architecture. Next, we describe a profile-guided data allocation algorithm that can increase the degree of I/O parallelism present in the system, as well as to balance I/O in a heterogeneous system. We present results on an actual implementation. Our experimental results show that by partitioning data across all available storage devices and carefully tuning I/O workloads in the grid system, our peer-to-peer scheme can deliver scalable high performance I/O that can address I/O-intensive workloads","PeriodicalId":255312,"journal":{"name":"2005 IEEE International Conference on Cluster Computing","volume":"41 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2005-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"9","resultStr":"{\"title\":\"Load Balancing using Grid-based Peer-to-Peer Parallel I/O\",\"authors\":\"Yijian Wang, D. Kaeli\",\"doi\":\"10.1109/CLUSTR.2005.347040\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"In the area of grid computing, there is a growing need to process large amounts of data. To support this trend, we need to develop efficient parallel storage systems that can provide for high performance for data-intensive applications. In order to overcome I/O bottlenecks and to increase I/O parallelism, data streams need to be parallelized at both the application level and the storage device level. In this paper, we propose a novel peer-to-peer (P2P) storage architecture for MPI applications on grid systems. We first present an analytic model of our P2P storage architecture. Next, we describe a profile-guided data allocation algorithm that can increase the degree of I/O parallelism present in the system, as well as to balance I/O in a heterogeneous system. We present results on an actual implementation. Our experimental results show that by partitioning data across all available storage devices and carefully tuning I/O workloads in the grid system, our peer-to-peer scheme can deliver scalable high performance I/O that can address I/O-intensive workloads\",\"PeriodicalId\":255312,\"journal\":{\"name\":\"2005 IEEE International Conference on Cluster Computing\",\"volume\":\"41 1\",\"pages\":\"0\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2005-09-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"9\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"2005 IEEE International Conference on Cluster Computing\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1109/CLUSTR.2005.347040\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"2005 IEEE International Conference on Cluster Computing","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/CLUSTR.2005.347040","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 9

摘要

在网格计算领域,处理大量数据的需求日益增长。为了支持这一趋势,我们需要开发能够为数据密集型应用程序提供高性能的高效并行存储系统。为了克服I/O瓶颈并提高I/O并行性,数据流需要在应用程序级别和存储设备级别并行化。在本文中,我们为网格系统上的MPI应用提出了一种新的点对点(P2P)存储架构。我们首先提出了P2P存储架构的分析模型。接下来,我们描述了一个配置文件引导的数据分配算法,该算法可以增加系统中存在的I/O并行度,并平衡异构系统中的I/O。我们给出了一个实际实现的结果。我们的实验结果表明,通过在所有可用的存储设备上划分数据并仔细调整网格系统中的I/O工作负载,我们的点对点方案可以提供可扩展的高性能I/O,可以解决I/O密集型工作负载
本文章由计算机程序翻译,如有差异,请以英文原文为准。
Load Balancing using Grid-based Peer-to-Peer Parallel I/O
In the area of grid computing, there is a growing need to process large amounts of data. To support this trend, we need to develop efficient parallel storage systems that can provide for high performance for data-intensive applications. In order to overcome I/O bottlenecks and to increase I/O parallelism, data streams need to be parallelized at both the application level and the storage device level. In this paper, we propose a novel peer-to-peer (P2P) storage architecture for MPI applications on grid systems. We first present an analytic model of our P2P storage architecture. Next, we describe a profile-guided data allocation algorithm that can increase the degree of I/O parallelism present in the system, as well as to balance I/O in a heterogeneous system. We present results on an actual implementation. Our experimental results show that by partitioning data across all available storage devices and carefully tuning I/O workloads in the grid system, our peer-to-peer scheme can deliver scalable high performance I/O that can address I/O-intensive workloads
求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信