Doopnet: An emulator for network performance analysis of Hadoop clusters using Docker and Mininet

2016 IEEE Symposium on Computers and Communication (ISCC) Pub Date : 2016-06-27 DOI:10.1109/ISCC.2016.7543832

Yuansong Qiao, Xueyuan Wang, Guiming Fang, Brian A. Lee

{"title":"Doopnet: An emulator for network performance analysis of Hadoop clusters using Docker and Mininet","authors":"Yuansong Qiao, Xueyuan Wang, Guiming Fang, Brian A. Lee","doi":"10.1109/ISCC.2016.7543832","DOIUrl":null,"url":null,"abstract":"Hadoop is one of the most important Big Data processing and storage systems. In recent years, a lot of efforts have been put to enhance Hadoop's performance from networking perspectives. However, there are limited tools that can help researchers to verify their networking algorithm design in terms of Hadoop's performance. This paper proposes Doopnet which is a framework and toolset for creating Hadoop clusters in a virtualized environment and for monitoring/analysing of Hadoop's networking characteristics under different network configurations. Doopnet enables users to automatically set up a Hadoop cluster over Docker containers running inside Mininet. The Hadoop traffic is collected inside the containers and virtual switches through network flow monitors. The users can easily modify network topologies or configurations through Mininet, observe the networking behaviour through network flow monitors, and analyse the effects of different network settings on Hadoop's performance. Examples are presented to demonstrate how to setup the Doopnet testbed and analyse Hadoop traffic.","PeriodicalId":148096,"journal":{"name":"2016 IEEE Symposium on Computers and Communication (ISCC)","volume":"22 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2016-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"10","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"2016 IEEE Symposium on Computers and Communication (ISCC)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/ISCC.2016.7543832","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}

引用次数: 10

Abstract

Hadoop is one of the most important Big Data processing and storage systems. In recent years, a lot of efforts have been put to enhance Hadoop's performance from networking perspectives. However, there are limited tools that can help researchers to verify their networking algorithm design in terms of Hadoop's performance. This paper proposes Doopnet which is a framework and toolset for creating Hadoop clusters in a virtualized environment and for monitoring/analysing of Hadoop's networking characteristics under different network configurations. Doopnet enables users to automatically set up a Hadoop cluster over Docker containers running inside Mininet. The Hadoop traffic is collected inside the containers and virtual switches through network flow monitors. The users can easily modify network topologies or configurations through Mininet, observe the networking behaviour through network flow monitors, and analyse the effects of different network settings on Hadoop's performance. Examples are presented to demonstrate how to setup the Doopnet testbed and analyse Hadoop traffic.

查看原文本刊更多论文

Doopnet:一个使用Docker和Mininet对Hadoop集群进行网络性能分析的模拟器

Hadoop是最重要的大数据处理和存储系统之一。近年来，人们做了很多努力来从网络的角度增强Hadoop的性能。然而，有有限的工具可以帮助研究人员在Hadoop的性能方面验证他们的网络算法设计。本文提出了Doopnet，它是一个框架和工具集，用于在虚拟化环境中创建Hadoop集群，并用于监控/分析Hadoop在不同网络配置下的网络特性。Doopnet使用户能够在Mininet内部运行的Docker容器上自动设置Hadoop集群。Hadoop流量是通过网络流量监视器在容器和虚拟交换机内部收集的。用户可以通过Mininet轻松修改网络拓扑或配置，通过网络流量监视器观察网络行为，分析不同网络设置对Hadoop性能的影响。举例说明了如何建立Doopnet测试平台和分析Hadoop流量。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

2016 IEEE Symposium on Computers and Communication (ISCC)

自引率

0.00%

发文量