Applying MapReduce Framework to Peer-to-Peer Overlay Network

Pei Xiao, Xiaolu Zhang, Jin Wang, Jixian Zhang, Qiang Han, Xuejie Zhang
{"title":"Applying MapReduce Framework to Peer-to-Peer Overlay Network","authors":"Pei Xiao, Xiaolu Zhang, Jin Wang, Jixian Zhang, Qiang Han, Xuejie Zhang","doi":"10.1109/ICSS.2014.21","DOIUrl":null,"url":null,"abstract":"MapReduce is a programming framework widely used in cloud computing environments for processing large amount of data in a highly parallel way. However, current MapReduce model do not cope well with its scalability, which means that under certain hardware configuration, it can only support limited scale of cluster due to the overloading of center node. In this paper, we present a prototype based on DHTs Peer-to-Peer MapReduce system, which removed the MapReduce task centralized scheduling's master node and bottom file system management's name node on the basis of remaining original MapReduce workflow unchanged. In the system, the distributed file system in bottom layer queries data through distributed hashing, while the MapReduce system in upper layer invoke and schedule the tasks by distributed notification mechanism. In this way, the system can theoretically achieve the scalability of Peer-to-Peer system. The scalability evaluation of the system has been experimented in the network scenarios using the prevailing word count problem.","PeriodicalId":206490,"journal":{"name":"2014 International Conference on Service Sciences","volume":"40 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2014-05-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"1","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"2014 International Conference on Service Sciences","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/ICSS.2014.21","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 1

Abstract

MapReduce is a programming framework widely used in cloud computing environments for processing large amount of data in a highly parallel way. However, current MapReduce model do not cope well with its scalability, which means that under certain hardware configuration, it can only support limited scale of cluster due to the overloading of center node. In this paper, we present a prototype based on DHTs Peer-to-Peer MapReduce system, which removed the MapReduce task centralized scheduling's master node and bottom file system management's name node on the basis of remaining original MapReduce workflow unchanged. In the system, the distributed file system in bottom layer queries data through distributed hashing, while the MapReduce system in upper layer invoke and schedule the tasks by distributed notification mechanism. In this way, the system can theoretically achieve the scalability of Peer-to-Peer system. The scalability evaluation of the system has been experimented in the network scenarios using the prevailing word count problem.
MapReduce框架在p2p覆盖网络中的应用
MapReduce是一种广泛应用于云计算环境的编程框架,用于以高度并行的方式处理大量数据。然而,目前的MapReduce模型并不能很好地应对它的可扩展性,这意味着在一定的硬件配置下,由于中心节点的过载,它只能支持有限的集群规模。本文提出了一个基于dht的p2p MapReduce系统原型,在保持原有MapReduce工作流不变的基础上,去掉了MapReduce任务集中调度的主节点和底层文件系统管理的名称节点。系统中底层的分布式文件系统通过分布式哈希方式查询数据,上层的MapReduce系统通过分布式通知机制调用和调度任务。这样,系统理论上可以实现点对点系统的可扩展性。系统的可扩展性评估已经在网络场景中使用流行的单词计数问题进行了实验。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信