Statistics-based Workload Modeling for MapReduce

2012 IEEE 26th International Parallel and Distributed Processing Symposium Workshops & PhD Forum Pub Date : 2012-05-21 DOI:10.1109/IPDPSW.2012.254

Hailong Yang, Zhongzhi Luan, Wenjun Li, D. Qian, Gang Guan

{"title":"Statistics-based Workload Modeling for MapReduce","authors":"Hailong Yang, Zhongzhi Luan, Wenjun Li, D. Qian, Gang Guan","doi":"10.1109/IPDPSW.2012.254","DOIUrl":null,"url":null,"abstract":"Large-scale data-intensive computing with MapReduce framework in Cloud is becoming pervasive for the core business of many academic, government, and industrial organizations. Hadoop is by far the most successful realization of MapReduce framework. While MapReduce is easy-to-use, efficient and reliable for data-intensive computations, the excessive configuration parameters in Hadoop cause unexpected challenges when running various workloads with Hadoop cluster effectively. Consequently, developers who have less experience with the Hadoop configuration system may devote a significant effort to write an application with poor performance, because they have no idea how these configurations would influence the performance, or they are not even aware that these configurations exist. In this paper, we propose a statistic analysis approach to identify the relationships among workload characteristics, Hadoop configurations and workload performance. Several non-intuitive relationships between workload characteristics and relative performance are revealed and the experimental results demonstrate that our regression models accurately predict the performance of MapReduce workloads under different Hadoop configurations.","PeriodicalId":378335,"journal":{"name":"2012 IEEE 26th International Parallel and Distributed Processing Symposium Workshops & PhD Forum","volume":"44 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2012-05-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"24","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"2012 IEEE 26th International Parallel and Distributed Processing Symposium Workshops & PhD Forum","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/IPDPSW.2012.254","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}

引用次数: 24

Abstract

Large-scale data-intensive computing with MapReduce framework in Cloud is becoming pervasive for the core business of many academic, government, and industrial organizations. Hadoop is by far the most successful realization of MapReduce framework. While MapReduce is easy-to-use, efficient and reliable for data-intensive computations, the excessive configuration parameters in Hadoop cause unexpected challenges when running various workloads with Hadoop cluster effectively. Consequently, developers who have less experience with the Hadoop configuration system may devote a significant effort to write an application with poor performance, because they have no idea how these configurations would influence the performance, or they are not even aware that these configurations exist. In this paper, we propose a statistic analysis approach to identify the relationships among workload characteristics, Hadoop configurations and workload performance. Several non-intuitive relationships between workload characteristics and relative performance are revealed and the experimental results demonstrate that our regression models accurately predict the performance of MapReduce workloads under different Hadoop configurations.

查看原文本刊更多论文

基于统计的MapReduce工作负载建模

在云端使用MapReduce框架的大规模数据密集型计算正在成为许多学术、政府和工业组织的核心业务。Hadoop是迄今为止MapReduce框架最成功的实现。虽然MapReduce在数据密集型计算中易于使用、高效可靠，但Hadoop中过多的配置参数会给Hadoop集群有效运行各种工作负载带来意想不到的挑战。因此，对Hadoop配置系统缺乏经验的开发人员可能会投入大量精力编写性能较差的应用程序，因为他们不知道这些配置将如何影响性能，或者他们甚至不知道这些配置的存在。在本文中，我们提出了一种统计分析方法来识别工作负载特征、Hadoop配置和工作负载性能之间的关系。揭示了工作负载特征与相对性能之间的一些非直观关系，实验结果表明，我们的回归模型准确地预测了不同Hadoop配置下MapReduce工作负载的性能。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

2012 IEEE 26th International Parallel and Distributed Processing Symposium Workshops & PhD Forum

自引率

0.00%

发文量