Proceedings of the 10th Workshop on Workflows in Support of Large-Scale Science最新文献

筛选
英文 中文
Contemporary challenges for data-intensive scientific workflow management systems 数据密集型科学工作流管理系统面临的当代挑战
Proceedings of the 10th Workshop on Workflows in Support of Large-Scale Science Pub Date : 2015-11-15 DOI: 10.1145/2822332.2822336
Ryan Mork, Paul Martin, Zhiming Zhao
{"title":"Contemporary challenges for data-intensive scientific workflow management systems","authors":"Ryan Mork, Paul Martin, Zhiming Zhao","doi":"10.1145/2822332.2822336","DOIUrl":"https://doi.org/10.1145/2822332.2822336","url":null,"abstract":"Data-intensive sciences now represent the forefront of current scientific computing. To handle this 'Big Data' focus, scientists demand enabling technologies that can adapt to the increasingly distributed, collaborative, and exploratory scientific milieu. However, how these challenges have changed the design requirements of scientific workflow management systems (SWMSs) has not been assessed. First, how scientists currently use SWMSs was determined through a comprehensive usage survey examining 1455 research publications from 2013 to July 31st, 2015. To understand how data-intensive scientists are producing impactful research, we further examined usage of two major research clouds, the Open Science Data Cloud (OSDC) and Cornell's Red Cloud. Here, we present a road map for SWMS development for data-intensive sciences. SWMSs are now needed that interconnect diverse software packages while enabling data exploration and multi-user interaction across distributed software and hardware environments.","PeriodicalId":430484,"journal":{"name":"Proceedings of the 10th Workshop on Workflows in Support of Large-Scale Science","volume":"11 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-11-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125080516","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 14
Orchestrating workflows over heterogeneous networking infrastructures: NEWT: a network edge workflow tool 在异构网络基础设施上编排工作流:NEWT:一个网络边缘工作流工具
Proceedings of the 10th Workshop on Workflows in Support of Large-Scale Science Pub Date : 2015-11-15 DOI: 10.1145/2822332.2822334
J. Macker, I. Taylor
{"title":"Orchestrating workflows over heterogeneous networking infrastructures: NEWT: a network edge workflow tool","authors":"J. Macker, I. Taylor","doi":"10.1145/2822332.2822334","DOIUrl":"https://doi.org/10.1145/2822332.2822334","url":null,"abstract":"This paper investigates the use of the workflow methodology for group communication applications in highly distributed and dynamic wireless networks. The need for distributed ad hoc wireless communication applications is commonplace in defense and emergency response situations and is also becoming of more interest for commercial and industrial applications. Within this space, there is a need for a reusable structured methodology, such as what workflows can provide, to be able to specify collaborative applications, distributed processing and logic to support a variety of deployment scenarios in a common way. However, collaborative applications in highly dynamic and potentially disruptive network scenarios have rather different requirements to the way workflows currently operate over distributed computing infrastructures for three main reasons. In this paper, we identify and address each of these issues and introduce a lightweight workflow system that provides a solution for such environments and provides a common mechanism to describe and execute distributed workflows in distributed, ad hoc environments.","PeriodicalId":430484,"journal":{"name":"Proceedings of the 10th Workshop on Workflows in Support of Large-Scale Science","volume":"2 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-11-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126980516","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 0
Dynamically reconfigurable workflows for time-critical applications 用于时间关键型应用程序的动态可重构工作流
Proceedings of the 10th Workshop on Workflows in Support of Large-Scale Science Pub Date : 2015-11-15 DOI: 10.1145/2822332.2822339
Kieran Evans, Andrew C. Jones, A. Preece, F. Quevedo, David H. Rogers, Irena Spasic, I. Taylor, V. Stankovski, S. Taherizadeh, Jernej Trnkoczy, G. Suciu, Victor Suciu, Paul Martin, Junchao Wang, Zhiming Zhao
{"title":"Dynamically reconfigurable workflows for time-critical applications","authors":"Kieran Evans, Andrew C. Jones, A. Preece, F. Quevedo, David H. Rogers, Irena Spasic, I. Taylor, V. Stankovski, S. Taherizadeh, Jernej Trnkoczy, G. Suciu, Victor Suciu, Paul Martin, Junchao Wang, Zhiming Zhao","doi":"10.1145/2822332.2822339","DOIUrl":"https://doi.org/10.1145/2822332.2822339","url":null,"abstract":"Cloud-based applications that depend on time-critical data processing or network throughput require the capability of reconfiguring their infrastructure on demand as and when conditions change. Although the ability to apply quality of service constraints on the current Cloud offering is limited, there are ongoing efforts to change this. One such effort is the European funded SWITCH project that aims to provide a programming model and toolkit to help programmers specify quality of service and quality of experience metrics of their distributed application and to provide the means to specify the reconfiguration actions which can be taken to maintain these requirements. In this paper, we present an approach to application reconfiguration by applying a workflow methodology to implement a prototype involving multiple reconfiguration scenarios of a distributed real-time social media analysis application, called Sentinel. We show that by using a lightweight RPC-based workflow approach, we can monitor a live application in real time and spawn dependency-based workflows to reconfigure the underlying Docker containers that implement the distributed components of the application. We propose to use this prototype as the basis for part of the SWITCH workbench, which will support more advanced programmable infrastructures.","PeriodicalId":430484,"journal":{"name":"Proceedings of the 10th Workshop on Workflows in Support of Large-Scale Science","volume":"5 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-11-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132267096","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 36
Enabling workflow repeatability with virtualization support 通过虚拟化支持实现工作流的可重复性
Proceedings of the 10th Workshop on Workflows in Support of Large-Scale Science Pub Date : 2015-11-15 DOI: 10.1145/2822332.2822340
Fan Jiang, C. Castillo, Charles Schmitt, A. Mandal, P. Ruth, I. Baldin
{"title":"Enabling workflow repeatability with virtualization support","authors":"Fan Jiang, C. Castillo, Charles Schmitt, A. Mandal, P. Ruth, I. Baldin","doi":"10.1145/2822332.2822340","DOIUrl":"https://doi.org/10.1145/2822332.2822340","url":null,"abstract":"The value of workflows to the scientific community spans over time and space. Not only results but also performance and resource consumption of a workflow need to be replayed over time and in varying environments. Achieving such repeatability in practice is challenging due to changes in software and infrastructure over time. In this work, we introduce a new abstraction that builds on the concept of virtual appliance to enable workflow repeatability. We have also developed a novel architecture to leverage this abstraction and realized it into a system implementation that supports a popular workflow management system and builds on a federated in-production environment. We demonstrate the effectiveness of our approach by examining various aspects of workflow repeatability. Our results show that workflows can be replayed with 2% fidelity when considering their walltime as performance metric.","PeriodicalId":430484,"journal":{"name":"Proceedings of the 10th Workshop on Workflows in Support of Large-Scale Science","volume":"41 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-11-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133194757","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 5
Proceedings of the 10th Workshop on Workflows in Support of Large-Scale Science 第十届支持大规模科学工作流程研讨会论文集
J. Montagnat, I. Taylor
{"title":"Proceedings of the 10th Workshop on Workflows in Support of Large-Scale Science","authors":"J. Montagnat, I. Taylor","doi":"10.1145/2822332","DOIUrl":"https://doi.org/10.1145/2822332","url":null,"abstract":"This year is special for the WORKS series as this corresponds to the tenth issue of this scientific event dedicated to scientific workflows. The call for papers attracted thirteen submissions from Europe, the USA, India and Mexico. After peer reviews by the program committee, nine of the papers were accepted, covering a variety of topics: workflows tasks scheduling, resources allocation for efficient execution, data flows management, languages, workflow adaptation, virtualization and provenance. \u0000 \u0000Looking over the past years, the problem of workflow scheduling and resources allocation in distributed infrastructures has always been well represented in the WORKS workshop series, with lan increasing interest for cloud type of resources lately. Together with scalability issues, they have represented a fundamental research axis over a long period of time. Accompanying infrastructure virtualization, effort has also been recently invested in abstract workflow representations and dynamic execution processes allowing for reconfiguration depending on the evolving execution conditions or intermediate results found. Finally, it can be seen from this year program that data flow management is a topic with a renewed interest in the broader context of scientific Big Data analysis.","PeriodicalId":430484,"journal":{"name":"Proceedings of the 10th Workshop on Workflows in Support of Large-Scale Science","volume":"81 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-11-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126001362","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
Co-sites: the autonomous distributed dataflows in collaborative scientific discovery 协同站点:协同科学发现中的自主分布式数据流
Proceedings of the 10th Workshop on Workflows in Support of Large-Scale Science Pub Date : 2015-11-15 DOI: 10.1145/2822332.2822337
Yanwei Zhang, M. Wolf, K. Schwan, Qing Liu, G. Eisenhauer, S. Klasky
{"title":"Co-sites: the autonomous distributed dataflows in collaborative scientific discovery","authors":"Yanwei Zhang, M. Wolf, K. Schwan, Qing Liu, G. Eisenhauer, S. Klasky","doi":"10.1145/2822332.2822337","DOIUrl":"https://doi.org/10.1145/2822332.2822337","url":null,"abstract":"Online \"big data\" processing applications have seen increasing importance in the high performance computing domain, including online analytics of large volumes of data output by various scientific applications. This work contributes to answering the question of how to promote efficient collaborative science in face of unpredictable analytics workloads and dynamics in available resources? It proposes the Co-Sites solution employing online resource management at the sites participating online collaboration, including geographically distributed sites that may spread across large distances. Co-Sites operates by each site observing its local progress and making its own decisions to better utilize local resources and to maintain acceptable rates of global progress. Co-Sites further enriches such distributed data flows to permit just-in-time data sharing to better leverage collaborators' diverse domain expertise. Experiments with a combustion workflow demonstrate the Co-Sites solution with (i) improved end-to-end completion times, (ii) good scalability, and (iii) with good data sharing latencies.","PeriodicalId":430484,"journal":{"name":"Proceedings of the 10th Workshop on Workflows in Support of Large-Scale Science","volume":"109 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-11-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115815316","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
Towards efficient scheduling of data intensive high energy physics workflows 面向数据密集型高能物理工作流程的高效调度
Proceedings of the 10th Workshop on Workflows in Support of Large-Scale Science Pub Date : 2015-11-15 DOI: 10.1145/2822332.2822335
M. Halappanavar, M. Schram, L. D. L. Torre, K. Barker, Nathan R. Tallent, D. Kerbyson
{"title":"Towards efficient scheduling of data intensive high energy physics workflows","authors":"M. Halappanavar, M. Schram, L. D. L. Torre, K. Barker, Nathan R. Tallent, D. Kerbyson","doi":"10.1145/2822332.2822335","DOIUrl":"https://doi.org/10.1145/2822332.2822335","url":null,"abstract":"Data intensive high energy physics workflows executed on geographically distributed resources pose a tremendous challenge for efficient use of computing resources. In this early work paper, we present a hierarchical framework for efficient allocation of resources and energy-efficient assignment of tasks for a representative high energy physics application, the Belle II experiments. With an expected data rate of 25 peta bytes per year from experimental data and Monte Carlo simulations, the Belle II experiment provides an ideal platform for algorithmic development. Building on the analogy of the unit commitment problem in electric power grids, we present a novel cost-efficient method for resource allocation that feeds into energy-efficient assignment of tasks to resources using a novel semi-matching based algorithm. We demonstrate that this approach is both computationally efficient and effective. We expect the methods developed in this work to benefit Belle II and other complex workflows executed on distributed resources.","PeriodicalId":430484,"journal":{"name":"Proceedings of the 10th Workshop on Workflows in Support of Large-Scale Science","volume":"19 26","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-11-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131805994","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 10
Workflow provenance: an analysis of long term storage costs 工作流来源:对长期存储成本的分析
Proceedings of the 10th Workshop on Workflows in Support of Large-Scale Science Pub Date : 2015-11-15 DOI: 10.1145/2822332.2822341
S. Woodman, H. Hiden, P. Watson
{"title":"Workflow provenance: an analysis of long term storage costs","authors":"S. Woodman, H. Hiden, P. Watson","doi":"10.1145/2822332.2822341","DOIUrl":"https://doi.org/10.1145/2822332.2822341","url":null,"abstract":"The storage and retrieval of provenance is a critical piece of functionality for many data processing systems. There are numerous cases where, in order to satisfy regulatory requirements (such as drug development and medical data processing), accurately reproduce results (scientific research) or to maintain financial transparency (for example to meet Sarbanes Oxley regulations in the US), a full and accurate provenance trace is vital. Whilst it is always possible to meet these requirements by storing every piece of intermediate data generated by a sequence of calculations, the costs associated with retaining data that may have a low probability of future retrieval is significant. There is, however, an opportunity for a reduction in the cost of storage by opting not to store certain intermediate results that can be regenerated given a knowledge of the processing code and input data that generated them. This paper presents a approach which is able, via a collection of past performance and provenance data, to make decisions based on the underlying storage and computation costs as to which intermediate data to retain and which to regenerate on demand.","PeriodicalId":430484,"journal":{"name":"Proceedings of the 10th Workshop on Workflows in Support of Large-Scale Science","volume":"50 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-11-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130365602","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 11
A workflow runtime environment for manycore parallel architectures 多核并行架构的工作流运行时环境
Proceedings of the 10th Workshop on Workflows in Support of Large-Scale Science Pub Date : 2015-11-15 DOI: 10.1145/2822332.2822333
Matthias Janetschek, R. Prodan, S. Benedict
{"title":"A workflow runtime environment for manycore parallel architectures","authors":"Matthias Janetschek, R. Prodan, S. Benedict","doi":"10.1145/2822332.2822333","DOIUrl":"https://doi.org/10.1145/2822332.2822333","url":null,"abstract":"We introduce a new Manycore Workflow Runtime Environment (MWRE) to efficiently enact traditional scientific workflows on modern manycore computing architectures. In contrast to existing engines that enact workflows acting as external services, MWRE is compiler-based and translates workflows specified in the XML-based Interoperable Workflow Intermediate Representation (IWIR) into an equivalent C++-based program. This program efficiently enacts the workflow as a stand-alone executable by means of a new callback mechanism that resolves dependencies, transfers data, and handles composite activities. Experimental results on a number of real-world workflows demonstrate that MWRE clearly outperforms existing Java-based workflow engines designed for distributed (Grid/Cloud) computing infrastructures in terms of enactment time, is generally better than an existing script-based engine for manycore architectures (Swift), and sometimes gets even close to an artificial baseline implementation of the workflows in the standard OpenMP language for shared memory systems.","PeriodicalId":430484,"journal":{"name":"Proceedings of the 10th Workshop on Workflows in Support of Large-Scale Science","volume":"15 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-11-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126548969","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 11
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
相关产品
×
本文献相关产品
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信
小红书