绿色存储在Hadoop中的实现

Proceedings of 2014 International Conference on Cloud Computing and Internet of Things Pub Date : 2014-12-01 DOI:10.1109/CCIOT.2014.7062512

Qiao Zhu, Li Miao

{"title":"绿色存储在Hadoop中的实现","authors":"Qiao Zhu, Li Miao","doi":"10.1109/CCIOT.2014.7062512","DOIUrl":null,"url":null,"abstract":"Hadoop has been successful at harnessing expansive data-centers resources for large-scale data analysis. However, their effect on data-centers energy efficiency has not scrutinized completely. The energy consumption of Hadoop Distributed File System in data-centers accounts for a great part of total cost of ownership and disk is the main storage media for clusters. Analysis of the interactions between clusters with disks when running a Hadoop application showed a disk idle time when shuffle to memory, which can be used to guide a simple green storage algorithm for Hadoop cluster. The algorithm simulation results with Terasort for 10G data in a cluster with 11 nodes can save 2.47WH.","PeriodicalId":255477,"journal":{"name":"Proceedings of 2014 International Conference on Cloud Computing and Internet of Things","volume":"44 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2014-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"2","resultStr":"{\"title\":\"The realization of green storage in Hadoop\",\"authors\":\"Qiao Zhu, Li Miao\",\"doi\":\"10.1109/CCIOT.2014.7062512\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Hadoop has been successful at harnessing expansive data-centers resources for large-scale data analysis. However, their effect on data-centers energy efficiency has not scrutinized completely. The energy consumption of Hadoop Distributed File System in data-centers accounts for a great part of total cost of ownership and disk is the main storage media for clusters. Analysis of the interactions between clusters with disks when running a Hadoop application showed a disk idle time when shuffle to memory, which can be used to guide a simple green storage algorithm for Hadoop cluster. The algorithm simulation results with Terasort for 10G data in a cluster with 11 nodes can save 2.47WH.\",\"PeriodicalId\":255477,\"journal\":{\"name\":\"Proceedings of 2014 International Conference on Cloud Computing and Internet of Things\",\"volume\":\"44 1\",\"pages\":\"0\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2014-12-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"2\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Proceedings of 2014 International Conference on Cloud Computing and Internet of Things\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1109/CCIOT.2014.7062512\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Proceedings of 2014 International Conference on Cloud Computing and Internet of Things","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/CCIOT.2014.7062512","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}

引用次数: 2

摘要

Hadoop在利用扩展的数据中心资源进行大规模数据分析方面取得了成功。然而，它们对数据中心能源效率的影响还没有得到完全的研究。Hadoop分布式文件系统在数据中心的能耗占总拥有成本的很大一部分，磁盘是集群的主要存储介质。分析运行Hadoop应用程序时集群与磁盘之间的交互，显示shuffle到内存时磁盘空闲时间，可用于指导Hadoop集群的简单绿色存储算法。采用Terasort对11个节点集群中10G数据进行算法仿真，可节省2.47WH。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

查看原文本刊更多论文

The realization of green storage in Hadoop

Hadoop has been successful at harnessing expansive data-centers resources for large-scale data analysis. However, their effect on data-centers energy efficiency has not scrutinized completely. The energy consumption of Hadoop Distributed File System in data-centers accounts for a great part of total cost of ownership and disk is the main storage media for clusters. Analysis of the interactions between clusters with disks when running a Hadoop application showed a disk idle time when shuffle to memory, which can be used to guide a simple green storage algorithm for Hadoop cluster. The algorithm simulation results with Terasort for 10G data in a cluster with 11 nodes can save 2.47WH.

求助全文

通过发布文献求助，成功后即可免费获取论文全文。去求助

来源期刊

Proceedings of 2014 International Conference on Cloud Computing and Internet of Things

自引率

0.00%

发文量