SOR-HDFS:基于seda的方法，在rdma增强的HDFS中最大化重叠

IEEE International Symposium on High-Performance Parallel Distributed Computing Pub Date : 2014-06-23 DOI:10.1145/2600212.2600715

Nusrat S. Islam, Xiaoyi Lu, Md. Wasi-ur-Rahman, D. Panda

{"title":"SOR-HDFS:基于seda的方法，在rdma增强的HDFS中最大化重叠","authors":"Nusrat S. Islam, Xiaoyi Lu, Md. Wasi-ur-Rahman, D. Panda","doi":"10.1145/2600212.2600715","DOIUrl":null,"url":null,"abstract":"In this paper, we propose SOR-HDFS, a SEDA (Staged Event-Driven Architecture)-based approach to improve the performance of HDFS Write operation. This design not only incorporates RDMA-based communication over InfiniBand but also maximizes overlapping among different stages of data transfer and I/O. Performance evaluations show that, the new design improves the aggregated write throughput of Enhanced DFSIO benchmark in Intel HiBench by up to 64% and reduces the job execution time by 37% compared to IPoIB (IP over InfiniBand). Compared to the previous best RDMA-enhanced design [4], the improvements in throughput and execution time are 30% and 20%, respectively. Our design can also improve the performance of HBase Put operation by up to 53% over IPoIB and 29% compared to the previous best RDMA-enhanced HDFS. To the best of our knowledge, this is the first design of SEDA-based HDFS in the literature.","PeriodicalId":330072,"journal":{"name":"IEEE International Symposium on High-Performance Parallel Distributed Computing","volume":"5 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2014-06-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"31","resultStr":"{\"title\":\"SOR-HDFS: a SEDA-based approach to maximize overlapping in RDMA-enhanced HDFS\",\"authors\":\"Nusrat S. Islam, Xiaoyi Lu, Md. Wasi-ur-Rahman, D. Panda\",\"doi\":\"10.1145/2600212.2600715\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"In this paper, we propose SOR-HDFS, a SEDA (Staged Event-Driven Architecture)-based approach to improve the performance of HDFS Write operation. This design not only incorporates RDMA-based communication over InfiniBand but also maximizes overlapping among different stages of data transfer and I/O. Performance evaluations show that, the new design improves the aggregated write throughput of Enhanced DFSIO benchmark in Intel HiBench by up to 64% and reduces the job execution time by 37% compared to IPoIB (IP over InfiniBand). Compared to the previous best RDMA-enhanced design [4], the improvements in throughput and execution time are 30% and 20%, respectively. Our design can also improve the performance of HBase Put operation by up to 53% over IPoIB and 29% compared to the previous best RDMA-enhanced HDFS. To the best of our knowledge, this is the first design of SEDA-based HDFS in the literature.\",\"PeriodicalId\":330072,\"journal\":{\"name\":\"IEEE International Symposium on High-Performance Parallel Distributed Computing\",\"volume\":\"5 1\",\"pages\":\"0\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2014-06-23\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"31\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"IEEE International Symposium on High-Performance Parallel Distributed Computing\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1145/2600212.2600715\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"IEEE International Symposium on High-Performance Parallel Distributed Computing","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1145/2600212.2600715","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}

引用次数: 31

摘要

在本文中，我们提出了基于SEDA (stage Event-Driven Architecture)的SOR-HDFS方法来提高HDFS写操作的性能。这种设计不仅结合了基于rdma的InfiniBand通信，而且最大限度地提高了数据传输和I/O不同阶段之间的重叠。性能评估表明，与IPoIB (IP over InfiniBand)相比，新设计将英特尔HiBench中Enhanced DFSIO基准测试的总写吞吐量提高了64%，并将作业执行时间缩短了37%。与之前最佳的rdma增强设计[4]相比，吞吐量和执行时间分别提高了30%和20%。我们的设计还可以将HBase的Put操作性能比IPoIB提高53%，比之前最好的rdma增强HDFS提高29%。据我们所知，这是文献中第一个基于seda的HDFS设计。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

查看原文本刊更多论文

SOR-HDFS: a SEDA-based approach to maximize overlapping in RDMA-enhanced HDFS

In this paper, we propose SOR-HDFS, a SEDA (Staged Event-Driven Architecture)-based approach to improve the performance of HDFS Write operation. This design not only incorporates RDMA-based communication over InfiniBand but also maximizes overlapping among different stages of data transfer and I/O. Performance evaluations show that, the new design improves the aggregated write throughput of Enhanced DFSIO benchmark in Intel HiBench by up to 64% and reduces the job execution time by 37% compared to IPoIB (IP over InfiniBand). Compared to the previous best RDMA-enhanced design [4], the improvements in throughput and execution time are 30% and 20%, respectively. Our design can also improve the performance of HBase Put operation by up to 53% over IPoIB and 29% compared to the previous best RDMA-enhanced HDFS. To the best of our knowledge, this is the first design of SEDA-based HDFS in the literature.

求助全文

通过发布文献求助，成功后即可免费获取论文全文。去求助

来源期刊

IEEE International Symposium on High-Performance Parallel Distributed Computing

自引率

0.00%

发文量