To Migrate or Not to Migrate: An Analysis of Operator Migration in Distributed Stream Processing

IF 34.4 1区计算机科学 Q1 COMPUTER SCIENCE, INFORMATION SYSTEMS

IEEE Communications Surveys and Tutorials Pub Date : 2023-11-07 DOI:10.1109/COMST.2023.3330953

Espen Volnes;Thomas Plagemann;Vera Goebel

{"title":"To Migrate or Not to Migrate: An Analysis of Operator Migration in Distributed Stream Processing","authors":"Espen Volnes;Thomas Plagemann;Vera Goebel","doi":"10.1109/COMST.2023.3330953","DOIUrl":null,"url":null,"abstract":"One of the most important issues in distributed data stream processing systems is using operator migration to handle highly variable workloads cost-efficiently and adapt to the needs at any given time on demand. Operator migration is a complex process involving changes in the state and stream management of a running query, typically without any data loss, and with as little disruption to the execution as possible. This tutorial aims to introduce operator migration, explain the core elements of operator migration, and provide the reader with a good understanding of the design alternatives used in existing solutions. We developed a conceptual model to explain the fundamentals of operator migration and introduce a unified terminology, leading to a taxonomy of existing solutions. The conceptual model separates mechanisms, i.e., how to migrate, and policy, i.e., when to migrate. This separation is further applied to structure the description of existing solutions, offering the reader an algorithmic perspective on various design alternatives. To enhance our understanding of the impact of various design alternatives on migration mechanisms, we also conducted an empirical study that provides quantitative insights. The operator downtime for the naïve migration approach is almost 20 times longer than when applying an incremental checkpoint-based approach.","PeriodicalId":55029,"journal":{"name":"IEEE Communications Surveys and Tutorials","volume":"26 1","pages":"670-705"},"PeriodicalIF":34.4000,"publicationDate":"2023-11-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10310197","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"IEEE Communications Surveys and Tutorials","FirstCategoryId":"94","ListUrlMain":"https://ieeexplore.ieee.org/document/10310197/","RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, INFORMATION SYSTEMS","Score":null,"Total":0}

引用次数: 0

Abstract

One of the most important issues in distributed data stream processing systems is using operator migration to handle highly variable workloads cost-efficiently and adapt to the needs at any given time on demand. Operator migration is a complex process involving changes in the state and stream management of a running query, typically without any data loss, and with as little disruption to the execution as possible. This tutorial aims to introduce operator migration, explain the core elements of operator migration, and provide the reader with a good understanding of the design alternatives used in existing solutions. We developed a conceptual model to explain the fundamentals of operator migration and introduce a unified terminology, leading to a taxonomy of existing solutions. The conceptual model separates mechanisms, i.e., how to migrate, and policy, i.e., when to migrate. This separation is further applied to structure the description of existing solutions, offering the reader an algorithmic perspective on various design alternatives. To enhance our understanding of the impact of various design alternatives on migration mechanisms, we also conducted an empirical study that provides quantitative insights. The operator downtime for the naïve migration approach is almost 20 times longer than when applying an incremental checkpoint-based approach.

查看原文本刊更多论文

迁移还是不迁移？分布式流处理中的操作员迁移分析

分布式数据流处理系统中最重要的问题之一是利用操作员迁移来经济高效地处理高度可变的工作负载，并随时按需适应各种需求。操作符迁移是一个复杂的过程，涉及更改正在运行的查询的状态和流管理，通常不会丢失任何数据，并尽可能不中断执行。本教程旨在介绍操作符迁移，解释操作符迁移的核心要素，并让读者充分了解现有解决方案中使用的设计方案。我们开发了一个概念模型来解释运算符迁移的基本原理，并引入统一的术语，从而对现有解决方案进行分类。概念模型将机制（即如何迁移）和策略（即何时迁移）分开。这种分离被进一步应用于现有解决方案的结构化描述，为读者提供了各种设计方案的算法视角。为了加深我们对各种设计方案对迁移机制影响的理解，我们还进行了一项实证研究，以提供定量见解。与采用基于增量检查点的方法相比，天真迁移方法的操作员停机时间几乎长了 20 倍。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

IEEE Communications Surveys and Tutorials COMPUTER SCIENCE, INFORMATION SYSTEMS-TELECOMMUNICATIONS

CiteScore

80.20

自引率

2.50%

发文量

审稿时长

6 months

期刊介绍： IEEE Communications Surveys & Tutorials is an online journal published by the IEEE Communications Society for tutorials and surveys covering all aspects of the communications field. Telecommunications technology is progressing at a rapid pace, and the IEEE Communications Society is committed to providing researchers and other professionals the information and tools to stay abreast. IEEE Communications Surveys and Tutorials focuses on integrating and adding understanding to the existing literature on communications, putting results in context. Whether searching for in-depth information about a familiar area or an introduction into a new area, IEEE Communications Surveys & Tutorials aims to be the premier source of peer-reviewed, comprehensive tutorials and surveys, and pointers to further sources. IEEE Communications Surveys & Tutorials publishes only articles exclusively written for IEEE Communications Surveys & Tutorials and go through a rigorous review process before their publication in the quarterly issues. A tutorial article in the IEEE Communications Surveys & Tutorials should be designed to help the reader to become familiar with and learn something specific about a chosen topic. In contrast, the term survey, as applied here, is defined to mean a survey of the literature. A survey article in IEEE Communications Surveys & Tutorials should provide a comprehensive review of developments in a selected area, covering its development from its inception to its current state and beyond, and illustrating its development through liberal citations from the literature. Both tutorials and surveys should be tutorial in nature and should be written in a style comprehensible to readers outside the specialty of the article.