SWEET '12 Pub Date : 2012-05-20 DOI: 10.1145/2443416.2443419

Thomas Tschager, H. Schmidt

引用次数: 5

Turbine: a distributed-memory dataflow engine for extreme-scale many-task applications Turbine:一个分布式内存数据流引擎，用于极端规模的多任务应用程序

SWEET '12 Pub Date : 2012-05-20 DOI: 10.1145/2443416.2443421

J. Wozniak, Timothy G. Armstrong, K. Maheshwari, E. Lusk, D. Katz, M. Wilde, Ian T Foster

引用次数: 32

Makeflow: a portable abstraction for data intensive computing on clusters, clouds, and grids makflow:用于集群、云和网格上的数据密集型计算的可移植抽象

SWEET '12 Pub Date : 2012-05-20 DOI: 10.1145/2443416.2443417

M. Albrecht, P. Donnelly, Peter Bui, D. Thain

{"title":"Makeflow: a portable abstraction for data intensive computing on clusters, clouds, and grids","authors":"M. Albrecht, P. Donnelly, Peter Bui, D. Thain","doi":"10.1145/2443416.2443417","DOIUrl":"https://doi.org/10.1145/2443416.2443417","url":null,"abstract":"In recent years, there has been a renewed interest in languages and systems for large scale distributed computing. Unfortunately, most systems available to the end user use a custom description language tightly coupled to a specific runtime implementation, making it difficult to transfer applications between systems. To address this problem we introduce Makeflow, a simple system for expressing and running a data-intensive workflow across multiple execution engines without requiring changes to the application or workflow description. Makeflow allows any user familiar with basic Unix Make syntax to generate a workflow and run it on one of many supported execution systems. Furthermore, in order to assess the performance characteristics of the various execution engines available to users and assist them in selecting one for use we introduce Workbench, a suite of benchmarks designed for analyzing common workflow patterns. We evaluate Workbench on two physical architectures -- the first a storage cluster with local disks and a slower network and the second a high performance computing cluster with a central parallel filesystem and fast network -- using a variety of execution engines. We conclude by demonstrating three applications that use Makeflow to execute data intensive applications consisting of thousands of jobs.","PeriodicalId":143151,"journal":{"name":"SWEET '12","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2012-05-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129902587","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 145

Evaluating parameter sweep workflows in high performance computing 高性能计算中参数扫描工作流的评估

SWEET '12 Pub Date : 2012-05-20 DOI: 10.1145/2443416.2443418

F. Chirigati, V. S. Sousa, Eduardo S. Ogasawara, Daniel de Oliveira, Jonas Dias, F. Porto, P. Valduriez, M. Mattoso

{"title":"Evaluating parameter sweep workflows in high performance computing","authors":"F. Chirigati, V. S. Sousa, Eduardo S. Ogasawara, Daniel de Oliveira, Jonas Dias, F. Porto, P. Valduriez, M. Mattoso","doi":"10.1145/2443416.2443418","DOIUrl":"https://doi.org/10.1145/2443416.2443418","url":null,"abstract":"Scientific experiments based on computer simulations can be defined, executed and monitored using Scientific Workflow Management Systems (SWfMS). Several SWfMS are available, each with a different goal and a different engine. Due to the exploratory analysis, scientists need to run parameter sweep (PS) workflows, which are workflows that are invoked repeatedly using different input data. These workflows generate a large amount of tasks that are submitted to High Performance Computing (HPC) environments. Different execution models for a workflow may have significant differences in performance in HPC. However, selecting the best execution model for a given workflow is difficult due to the existence of many characteristics of the workflow that may affect the parallel execution. We developed a study to show performance impacts of using different execution models in running PS workflows in HPC. Our study contributes by presenting a characterization of PS workflow patterns (the basis for many existing scientific workflows) and its behavior under different execution models in HPC. We evaluated four execution models to run workflows in parallel. Our study measures the performance behavior of small, large and complex workflows among the evaluated execution models. The results can be used as a guideline to select the best model for a given scientific workflow execution in HPC. Our evaluation may also serve as a basis for workflow designers to analyze the expected behavior of an HPC workflow engine based on the characteristics of PS workflows.","PeriodicalId":143151,"journal":{"name":"SWEET '12","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2012-05-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124470176","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 15

Oozie: towards a scalable workflow management system for Hadoop Oozie:面向Hadoop的可扩展工作流管理系统

SWEET '12 Pub Date : 2012-05-20 DOI: 10.1145/2443416.2443420

Mohammad Islam, Angelo K. Huang, Mohamed Battisha, Michelle Chiang, Santhosh Srinivasan, Craig Peters, A. Neumann, A. Abdelnur

引用次数: 107

SWEET '12最新文献