2011 IEEE International Conference on Cluster Computing最新文献

筛选

英文中文

Can a Decentralized Metadata Service Layer Benefit Parallel Filesystems? 去中心化的元数据服务层能使并行文件系统受益吗?

2011 IEEE International Conference on Cluster Computing Pub Date : 2011-09-01 DOI: 10.1109/CLUSTER.2011.85

V. Meshram, Xavier Besseron, Xiangyong Ouyang, R. Rajachandrasekar, R. Prakash, D. Panda

{"title":"Can a Decentralized Metadata Service Layer Benefit Parallel Filesystems?","authors":"V. Meshram, Xavier Besseron, Xiangyong Ouyang, R. Rajachandrasekar, R. Prakash, D. Panda","doi":"10.1109/CLUSTER.2011.85","DOIUrl":"https://doi.org/10.1109/CLUSTER.2011.85","url":null,"abstract":"The demand for scalable I/O continues to grow rapidly as computer clusters keep growing. Much of the research in storage systems has been focused on improving the scale and performance of I/O throughput. Scalable file systems do a good job of scaling large file access bandwidth by striping or sharing I/O resources across many servers or disks. However, the same cannot be said about scaling file metadata operation rates. Most existing parallel file systems choose to concentrate all the metadata processing load on a single server. This centralized processing can guarantee the correctness, but it severely hampers scalability. This downside is becoming more and more unacceptable as metadata throughput is critical for large scale applications. Distributing metadata processing load is critical to improve metadata scalability when handling huge number of client nodes. However, a solution to speed up metadata operations has to address two challenges simultaneously, namely the scalability and reliability. In this paper, we have designed a decentralized metadata service layer and evaluated its benefits and shortcomings that concern parallel file systems. The main aim of this service layer is to maintain reliability and consistency in a distributed metadata environment. At the same time we also focus on improving the scalability of the metadata operations, and in turn, the scalability of the underlying parallel file system. As demonstrated by experiments, the approach presented in this paper achieves significant improvements over native parallel file systems by large margin for all the major metadata operations. With 256 client processes, our decentralized metadata service outperforms Lustre and PVFS2 by a factor of 1.9 and 23, respectively, to create directories. With respect to stat() operation on files, our approach is 1.3 and 3.0 times faster than Lustre and PVFS.","PeriodicalId":200830,"journal":{"name":"2011 IEEE International Conference on Cluster Computing","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2011-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124344680","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 10

Play It Again, SimMR! 再来一遍，SimMR!

2011 IEEE International Conference on Cluster Computing Pub Date : 2011-08-31 DOI: 10.1109/CLUSTER.2011.36

Abhishek Verma, L. Cherkasova, R. Campbell

{"title":"Play It Again, SimMR!","authors":"Abhishek Verma, L. Cherkasova, R. Campbell","doi":"10.1109/CLUSTER.2011.36","DOIUrl":"https://doi.org/10.1109/CLUSTER.2011.36","url":null,"abstract":"A typical MapReduce cluster is shared among different users and multiple applications. A challenging problem in such shared environments is the ability to efficiently control resource allocations among the running and submitted jobs for achieving users' performance goals. To ease the task of evaluating and comparing different provisioning and scheduling approaches in MapReduce environments, we designed and implemented a simulation environment Sim MR which is comprised of three inter-related components: i) Trace Generator that creates a replayable MapReduce workload, ii) Simulator Engine that accurately emulates the job master functionality in Hadoop, and iii) a pluggable scheduling policy that dictates the scheduler decisions on job ordering and the amount of resources allocated to different jobs over time. We validate the accuracy of Sim MR environment by, first, executing a set of realistic MapReduce applications in a 66-node Hadoop cluster and then by replaying the collected job execution traces in SimMR. Our simulator accurately reproduces the original job processing: the completion times of the simulated jobs are within 5% of the original ones. SimMR can process over one million events per second. This allows users to simulate complex workloads in a few seconds instead of multi-hour executions in the real test bed. Finally, by using SimMR we analyze and compare performance of two novel deadline-driven schedulers over a diverse set of real and synthetic workloads.","PeriodicalId":200830,"journal":{"name":"2011 IEEE International Conference on Cluster Computing","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2011-08-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129757403","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 80

首页上一页