MC framework: High-performance distributed framework for standalone data analysis packages over Hadoop-based cloud

Chao-Chun Chen, Nguyen Huu Tinh Giang, Tzu-Chao Lin, Min-Hsiung Hung
{"title":"MC framework: High-performance distributed framework for standalone data analysis packages over Hadoop-based cloud","authors":"Chao-Chun Chen, Nguyen Huu Tinh Giang, Tzu-Chao Lin, Min-Hsiung Hung","doi":"10.1109/GrC.2013.6740375","DOIUrl":null,"url":null,"abstract":"The Hadoop MapReduce is the programming model of designing the scalable distributed computing applications, that provides developers can attain automatic parallelization. However, most complex manufacturing systems are arduous and restrictive to migrate to private clouds, due to the platform incompatible and tremendous complexity of system reconstruction. For increasing the efficiency of manufacturing systems with minimum efforts on modifying source codes, a high-performance framework is designed in this paper, called Multi-users-based Cloud-Adaptor Framework (MC-Framework), which provides the simple interface to users for fairly executing requested tasks worked with traditional standalone data analysis packages in MapReduce-based private cloud environments. Moreover, this framework focuses on multiuser workloads, but the default Hadoop scheduling scheme, i.e., FIFO, would increase delay under multiuser scenarios. Hence, a new scheduling mechanism, called Job-Sharing Scheduling, is designed to explore and fairly share the jobs to machines in the private cloud. Then, we prototype an experimental virtual-metrology module of a manufacturing system as a case study to verify and analysis the proposed MC-Framework. The results of our experiments indicate that our proposed framework enormously improved the time performance compared with the original package.","PeriodicalId":415445,"journal":{"name":"2013 IEEE International Conference on Granular Computing (GrC)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"2","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"2013 IEEE International Conference on Granular Computing (GrC)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/GrC.2013.6740375","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 2

Abstract

The Hadoop MapReduce is the programming model of designing the scalable distributed computing applications, that provides developers can attain automatic parallelization. However, most complex manufacturing systems are arduous and restrictive to migrate to private clouds, due to the platform incompatible and tremendous complexity of system reconstruction. For increasing the efficiency of manufacturing systems with minimum efforts on modifying source codes, a high-performance framework is designed in this paper, called Multi-users-based Cloud-Adaptor Framework (MC-Framework), which provides the simple interface to users for fairly executing requested tasks worked with traditional standalone data analysis packages in MapReduce-based private cloud environments. Moreover, this framework focuses on multiuser workloads, but the default Hadoop scheduling scheme, i.e., FIFO, would increase delay under multiuser scenarios. Hence, a new scheduling mechanism, called Job-Sharing Scheduling, is designed to explore and fairly share the jobs to machines in the private cloud. Then, we prototype an experimental virtual-metrology module of a manufacturing system as a case study to verify and analysis the proposed MC-Framework. The results of our experiments indicate that our proposed framework enormously improved the time performance compared with the original package.
MC框架:用于基于hadoop的云上的独立数据分析包的高性能分布式框架
Hadoop MapReduce是设计可伸缩分布式计算应用程序的编程模型,它为开发人员提供了实现自动并行化的方法。然而,由于平台不兼容和系统重构的巨大复杂性,大多数复杂制造系统迁移到私有云是困难和受限的。为了在最少修改源代码的情况下提高制造系统的效率,本文设计了一个高性能框架,称为基于多用户的云适配器框架(MC-Framework),该框架为用户提供了简单的接口,以便在基于mapreduce的私有云环境中公平地执行与传统独立数据分析包一起工作的请求任务。此外,该框架侧重于多用户工作负载,但默认的Hadoop调度方案,即FIFO,会增加多用户场景下的延迟。因此,设计了一种新的调度机制,称为作业共享调度,用于探索并公平地将作业共享给私有云中的机器。然后,我们以一个制造系统的实验性虚拟计量模块为原型,作为案例研究来验证和分析所提出的mc框架。实验结果表明,与原始包相比,我们提出的框架大大提高了时间性能。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信