Water & sewage works最新文献

Optimizing bioinformatics workflows for data analysis using cloud management techniques 使用云管理技术优化数据分析的生物信息学工作流程

Water & sewage works Pub Date : 2011-11-14 DOI: 10.1145/2110497.2110503

Vincent C. Emeakaroha, Pawel P. Labaj, M. Maurer, I. Brandić, David P. Kreil

{"title":"Optimizing bioinformatics workflows for data analysis using cloud management techniques","authors":"Vincent C. Emeakaroha, Pawel P. Labaj, M. Maurer, I. Brandić, David P. Kreil","doi":"10.1145/2110497.2110503","DOIUrl":"https://doi.org/10.1145/2110497.2110503","url":null,"abstract":"With the rapid development in recent years of high-throughput technologies in the life sciences, huge amounts of data are being generated and stored in databases. Despite significant advances in computing capacity and performance, an analysis of these large-scale data in a search for biomedically relevant patterns remains a challenging task. Scientific workflow applications support data-mining in more complex scenarios that include many data sources and computational tools, as commonly found in bioinformatics. A scientific workflow application is a holistic unit that defines, executes, and manages scientific applications using different software tools. Existing workflow applications are process- or data- rather than resource-oriented. Thus, they lack efficient computational resource management capabilities, such as those provided by Cloud computing environments. Insufficient computational resources disrupt the execution of workflow applications, wasting time and money. To address this issue, advanced resource monitoring and management strategies are required to determine the resource consumption behaviours of workflow applications for a dynamical allocation and deallocation of resources. In this paper, we present a novel Cloud resource monitoring technique and a knowledge management strategy to manage computational resources for workflow applications in order to guarantee their performance goals and their successful completion. We present the design description of these techniques, demonstrate how they can be applied to scientific workflow applications, and present first evaluation results as a proof of concept.","PeriodicalId":87858,"journal":{"name":"Water & sewage works","volume":"91 1","pages":"37-46"},"PeriodicalIF":0.0,"publicationDate":"2011-11-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"83786234","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 17

Object reuse and exchange for publishing and sharing workflows 发布和共享工作流的对象重用和交换

Water & sewage works Pub Date : 2011-11-14 DOI: 10.1145/2110497.2110506

A. Harrison, Ian Harvey, Andrew C. Jones, David H. Rogers, I. Taylor

{"title":"Object reuse and exchange for publishing and sharing workflows","authors":"A. Harrison, Ian Harvey, Andrew C. Jones, David H. Rogers, I. Taylor","doi":"10.1145/2110497.2110506","DOIUrl":"https://doi.org/10.1145/2110497.2110506","url":null,"abstract":"The workflow paradigm can provide the means to describe the complete functional pipeline for a scientific experiment and therefore expose the underlying scientific processes for enabling the reproducibility of results. However, current means for exposing such information are tied closely to the individual workflow engines and there is no existing method that provides a common way to share this information.\u0000 In this paper, we discuss a lightweight approach that can be used to expose such information, using the Open Archives Initiative Object Reuse and Exchange (ORE) standard, to provide a common format for representing and sharing workflows and their associated metadata required for their execution.\u0000 We describe how workflows can be mapped to the ORE format using RDF and how they can be stored using bundles for sharing with others. We discuss tooling we have developed that provides a mechanism for existing workflow engines to conveniently export workflows as ORE bundles. We present three use cases for Triana, ASKALON and MOTEUR, where such integration has already been undertaken, and conclude the paper by providing a short study showing that the overhead implications of adopting the proposed ORE bundling format are minimal.","PeriodicalId":87858,"journal":{"name":"Water & sewage works","volume":"3 1","pages":"67-76"},"PeriodicalIF":0.0,"publicationDate":"2011-11-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"89881806","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 3

Making data analysis expertise broadly accessible through workflows 通过工作流程使数据分析专业知识广泛可用

Water & sewage works Pub Date : 2011-11-14 DOI: 10.1145/2110497.2110507

M. Hauder, Y. Gil, Ricky J. Sethi, Yan Liu, Hyunjoon Jo

引用次数: 26

Characterizing quality of resilience in scientific workflows 描述科学工作流程中弹性的质量特征

Water & sewage works Pub Date : 2011-11-14 DOI: 10.1145/2110497.2110511

Rafael Tolosana-Calasanz, M. Lackovic, O. Rana, J. A. Bañares, D. Talia

{"title":"Characterizing quality of resilience in scientific workflows","authors":"Rafael Tolosana-Calasanz, M. Lackovic, O. Rana, J. A. Bañares, D. Talia","doi":"10.1145/2110497.2110511","DOIUrl":"https://doi.org/10.1145/2110497.2110511","url":null,"abstract":"The enactment of scientific workflows involves the distribution of tasks to distributed resources that exist in different administrative domains. Such resources can range in granularity from a single machine to one or more clusters and file systems. The use of such distributed resources during workflow enactment can be an error prone process and may lead to faults, which can range in type, frequency of occurrence and complexity. However, the level of fault tolerance available within many existing workflow engines varies significantly, ranging from no support available (requiring the user to intervene) or re-execution of a workflow automatically when a fault is detected. Many scientific workflows have to operate over heterogeneous infrastructure in the presence of failures -- therefore dealing with such failures in a more coherent way, so that a similar set of techniques can be applied across workflow engines is an important challenge. In this paper, we extend the concept of Quality of Service (QoS) -- where particular performance constraints need to be adhered to, with the concept of Quality of Resilience (QoR) -- a metric used to assess how resilient workflow enactment is likely to be in the presence of failures. We believe such a metric can guide: (i) the formulation of a workflow -- as a user annotated Directed Acyclic Graph (DAG); (ii) subsequent enactment of it over available resources. We identify how QoR must be considered at different levels -- from a user, workflow enactor and resource management perspectives. We identify the architectural elements that may be included within scientific workflows to support QoR and demonstrate the use of QoR in the Weka4WS data mining workflow system.","PeriodicalId":87858,"journal":{"name":"Water & sewage works","volume":"1 1","pages":"117-126"},"PeriodicalIF":0.0,"publicationDate":"2011-11-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"90143824","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 7

Achieving reproducibility by combining provenance with service and workflow versioning 通过将来源与服务和工作流版本控制相结合来实现再现性

Water & sewage works Pub Date : 2011-11-14 DOI: 10.1145/2110497.2110512

S. Woodman, H. Hiden, P. Watson, P. Missier

{"title":"Achieving reproducibility by combining provenance with service and workflow versioning","authors":"S. Woodman, H. Hiden, P. Watson, P. Missier","doi":"10.1145/2110497.2110512","DOIUrl":"https://doi.org/10.1145/2110497.2110512","url":null,"abstract":"Capturing and exploiting provenance information is considered to be important across a range of scientific, medical, commercial and Web applications, including recent trends towards publishing provenance-rich, executable papers. This article shows how the range of useful questions that provenance can answer is greatly increased when it is encapsulated into a system that can store and execute both current and old versions of workflows and services. e- Science Central provides a scalable, secure cloud platform for application developers. They can use it to upload data -- for storage on the cloud -- and services, which can be written in a variety of languages. These services can then be combined through workflows which are enacted in the cloud to compute over the data. When a workflow runs, a complete provenance trace is recorded. This paper shows how this provenance trace, used in conjunction with the ability to execute old versions of services and workflows (rather than just the latest versions) can provide useful information that would otherwise not be possible, including the key ability to reproduce experiments and to compare the effects of old and new versions of services on computations.","PeriodicalId":87858,"journal":{"name":"Water & sewage works","volume":"95 1","pages":"127-136"},"PeriodicalIF":0.0,"publicationDate":"2011-11-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"85855516","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 40

Provenance opportunities for WS-VLAM: an exploration of an e-science and an e-business approach WS-VLAM的起源机会:对电子科学和电子商务方法的探索

Water & sewage works Pub Date : 2011-11-14 DOI: 10.1145/2110497.2110505

M. Gerhards, V. Sander, Torsten Matzerath, A. Belloum, Dmitry Vasunin, A. Benabdelkader

{"title":"Provenance opportunities for WS-VLAM: an exploration of an e-science and an e-business approach","authors":"M. Gerhards, V. Sander, Torsten Matzerath, A. Belloum, Dmitry Vasunin, A. Benabdelkader","doi":"10.1145/2110497.2110505","DOIUrl":"https://doi.org/10.1145/2110497.2110505","url":null,"abstract":"Scientific applications are frequently modeled as a workflow that is executed under the control of a workflow management system. One crucial requirement during the execution is the validation of the generated results and the traceability of the experiment execution path. The automated tracking and storage of provenance information during workflow execution could satisfy this requirement. To collect provenance data using the Grid-enabled scientific workflow management system WS-VLAM, experimentation was made with two different implementations of the provenance concepts. The first one adopts the Open Provenance Model (OPM) as basis to represent, store, and share scientific experiments metadata using the Provenance Layer Infrastructure for e-Science Resources (PLIER). The second one is the history-tracing XML (HisT) which was developed for e-Business provenance. HisT provides a specific model to store provenance data within layered XML documents, whereby each layer is related to one individual workflow task. This paper explores these two provenance models by using an example workflow application and describes how they are integrated into WS-VLAM including implementation details of the provenance architecture. It finally gives a comparison of the two different approaches with a special regard to provenance for human actors.","PeriodicalId":87858,"journal":{"name":"Water & sewage works","volume":"14 1","pages":"57-66"},"PeriodicalIF":0.0,"publicationDate":"2011-11-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"84551839","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 7

A new approach for publishing workflows: abstractions, standards, and linked data 发布工作流的新方法:抽象、标准和关联数据

Water & sewage works Pub Date : 2011-11-14 DOI: 10.1145/2110497.2110504

D. Garijo, Y. Gil

引用次数: 93

IWIR: a language enabling portability across grid workflow systems IWIR:一种支持跨网格工作流系统可移植性的语言

Water & sewage works Pub Date : 2011-11-14 DOI: 10.1145/2110497.2110509

Kassian Plankensteiner, J. Montagnat, R. Prodan

引用次数: 28

Supporting dynamic parameter sweep in adaptive and user-steered workflow 支持自适应和用户导向工作流的动态参数扫描

Water & sewage works Pub Date : 2011-11-14 DOI: 10.1145/2110497.2110502

Jonas Dias, Eduardo S. Ogasawara, Daniel de Oliveira, F. Porto, A. Coutinho, M. Mattoso

引用次数: 34

AME: an anyscale many-task computing engine AME:任意规模的多任务计算引擎

Water & sewage works Pub Date : 2011-11-14 DOI: 10.1145/2110497.2110513

Zhao Zhang, D. Katz, M. Ripeanu, M. Wilde, Ian T Foster

{"title":"AME: an anyscale many-task computing engine","authors":"Zhao Zhang, D. Katz, M. Ripeanu, M. Wilde, Ian T Foster","doi":"10.1145/2110497.2110513","DOIUrl":"https://doi.org/10.1145/2110497.2110513","url":null,"abstract":"Many-Task Computing (MTC) is a new application category that encompasses increasingly popular applications in biology, economics, and statistics. The high inter-task parallelism and data-intensive processing capabilities of these applications pose new challenges to existing supercomputer hardware-software stacks. These challenges include resource provisioning; task dispatching, dependency resolution, and load balancing; data management; and resilience.\u0000 This paper examines the characteristics of MTC applications which create these challenges, and identifies related gaps in the middleware that supports these applications on extreme-scale systems. Based on this analysis, we propose AME, an Anyscale MTC Engine, which addresses the scalability aspects of these gaps. We describe the AME framework and present performance results for both synthetic benchmarks and real applications. Our results show that AME's dispatching performance linearly scales up to 14,120 tasks/second on 16,384 cores with high efficiency. The overhead of the intermediate data management scheme does not increase significantly up to 16,384 cores. AME eliminates 73% of the file transfer between compute nodes and the global filesystem for the Montage astronomy application running on 2,048 cores. Our results indicate that AME scales well on today's petascale machines, and is a strong candidate for exascale machines.","PeriodicalId":87858,"journal":{"name":"Water & sewage works","volume":"8 1","pages":"137-146"},"PeriodicalIF":0.0,"publicationDate":"2011-11-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"83358234","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 14