MCD: Overcoming the Data Download Bottleneck in Data Centers

2013 IEEE Eighth International Conference on Networking, Architecture and Storage Pub Date : 2013-07-17 DOI:10.1109/NAS.2013.18

J. Kaiser, Dirk Meister, Viktor Gottfried, A. Brinkmann

{"title":"MCD: Overcoming the Data Download Bottleneck in Data Centers","authors":"J. Kaiser, Dirk Meister, Viktor Gottfried, A. Brinkmann","doi":"10.1109/NAS.2013.18","DOIUrl":null,"url":null,"abstract":"The data download problem in data centers describes the increasingly common task of coordinated loading of identical data to a large number of nodes. Data download is seen as a significant problem in exascale HPC applications. Uncoor-dinated reading from a central file server creates contention at the file server and its network interconnect. We propose and evaluation a reliable multicast based approach to solve the data download problem. The MCD system builds a logical multi-rooted tree based on the physical network topology and uses the logical view for a two-phase approach. In the first phase, the data is multicasted to all nodes. In the second phase, the logical tree is used for an efficient error-correction. We evaluate the approach against the Twitter's Murder, which is BitTorrent-based data download solution used to deploy code binaries to thousands of nodes. The evaluation features a simulation of up to 10,000 nodes and shows that MCD finishes the reliable data download significantly faster. The simulation results are finally validated using a real-world deployment of more than 100 nodes.","PeriodicalId":213334,"journal":{"name":"2013 IEEE Eighth International Conference on Networking, Architecture and Storage","volume":"25 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2013-07-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"2013 IEEE Eighth International Conference on Networking, Architecture and Storage","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/NAS.2013.18","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}

引用次数: 0

Abstract

The data download problem in data centers describes the increasingly common task of coordinated loading of identical data to a large number of nodes. Data download is seen as a significant problem in exascale HPC applications. Uncoor-dinated reading from a central file server creates contention at the file server and its network interconnect. We propose and evaluation a reliable multicast based approach to solve the data download problem. The MCD system builds a logical multi-rooted tree based on the physical network topology and uses the logical view for a two-phase approach. In the first phase, the data is multicasted to all nodes. In the second phase, the logical tree is used for an efficient error-correction. We evaluate the approach against the Twitter's Murder, which is BitTorrent-based data download solution used to deploy code binaries to thousands of nodes. The evaluation features a simulation of up to 10,000 nodes and shows that MCD finishes the reliable data download significantly faster. The simulation results are finally validated using a real-world deployment of more than 100 nodes.

查看原文本刊更多论文

MCD:克服数据中心的数据下载瓶颈

数据中心中的数据下载问题描述了将相同的数据协调加载到大量节点的日益常见的任务。数据下载被认为是百亿亿次高性能计算应用程序中的一个重要问题。从中央文件服务器读取不协调的数据会在文件服务器及其网络互连上产生争用。我们提出并评估了一种可靠的基于组播的数据下载方法。MCD系统基于物理网络拓扑结构构建逻辑多根树，采用逻辑视图两阶段法。在第一阶段，数据被组播到所有节点。在第二阶段，使用逻辑树进行有效的纠错。我们对Twitter的谋杀方法进行了评估，这是一种基于bittorrent的数据下载解决方案，用于将代码二进制文件部署到数千个节点。该评估模拟了多达10,000个节点，并表明MCD完成可靠数据下载的速度明显更快。仿真结果最终通过100多个节点的实际部署得到验证。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

2013 IEEE Eighth International Conference on Networking, Architecture and Storage

自引率

0.00%

发文量