Proceedings of the 16th ACM International Conference on Distributed and Event-Based Systems最新文献

{"title":"Deriving a realistic workload model to simulate high-volume financial data feeds for performance benchmarking","authors":"Vladimir Sladojević, S Frischbier, Alexander Echler, Mario Paic, Alessandro Margara","doi":"10.1145/3524860.3539653","DOIUrl":"https://doi.org/10.1145/3524860.3539653","url":null,"abstract":"Processing financial market data at scale and in real-time poses a set of unique challenges to event-driven architectures due to the volume, variety, velocity, and veracity of the enclosed information on top of other constraints. Reproducible stress tests at scale using configurable benchmarks are key to building and tuning suitable processing systems. Available benchmarks, however, lack realistic and configurable workload models for market data scenarios. In previous work we already addressed this gap by describing the specific challenges of processing financial data at scale and by introducing a modular open-source benchmarking framework. This paper makes two contributions to the ongoing challenge of building realistic benchmarks for the financial data processing domain by outlining: (a) a detailed statistical analysis of real-world financial market data feeds processed on a global scale by Infront Financial Technology GmbH; and (b) a simple workload model built on this analysis to simulate high-volume market data feeds with their distinctive characteristics to be used in benchmarks. We evaluate our model using the DEBS 2022 Grand Challenge data set Trading Data.","PeriodicalId":263415,"journal":{"name":"Proceedings of the 16th ACM International Conference on Distributed and Event-Based Systems","volume":"8 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126641796","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A multi-level caching architecture for stateful stream computation","authors":"Muhammed Tawfiqul Islam, Renata Borovica-Gajic, S. Karunasekera","doi":"10.1145/3524860.3539803","DOIUrl":"https://doi.org/10.1145/3524860.3539803","url":null,"abstract":"Stream processing is used for real-time applications that deal with large volumes, velocities, and varieties of data. Stream processing frameworks discretize continuous data streams to apply computations on smaller batches. For real-time stream-based data analytics algorithms, the intermediate states of computations might need to be retained in memory until the query is complete. Thus, a massive surge in memory demand needs to be satisfied to run these algorithms successfully. However, a worker/server node in a computing cluster may have limited memory capacity. In addition, multiple parallel processes might be running concurrently, sharing the primary memory. As a result, a streaming application might fail to run or complete due to a memory shortage. Although spilling state information to the disk can alleviate the problem by allowing the query to finish, it will cause significant performance overhead. An in-memory-based object store as the state backend will also perform poorly due to the added communications with the external object store and serializing/deserializing the objects. This paper proposes a multi-level caching architecture to mitigate the surge of memory demand from the processes running complex stateful streaming applications. The multiple levels of the cache span across the process heap space, in-memory distributed object store, and secondary storage. The objects/states required in the computation are always served from the fastest level of the cache to boost the application performance. We also provide a multi-level caching library in Java which can be used to implement scalable streaming algorithms. The underlying cache management completely abstracts the multi-level cache implementation from the application and handles seamless migration of states/objects across different levels of the cache. In addition, the multi-layer caching architecture is configurable (i.e., an application can choose to leave out a cache level.) The experimental results demonstrate that our proposed multi-level caching approach for state management can manage large computational windows and improve the performance of an actual streaming application up to three times compared to the in-memory object store-based state backend.","PeriodicalId":263415,"journal":{"name":"Proceedings of the 16th ACM International Conference on Distributed and Event-Based Systems","volume":"18 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122131115","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A high-performance processing system for monitoring stock market data stream","authors":"Kevin Li, D. Fernandez, David Klingler, Yuhan Gao, Jacob Rivera, Kia Teymourian","doi":"10.1145/3524860.3539651","DOIUrl":"https://doi.org/10.1145/3524860.3539651","url":null,"abstract":"High-performance real-time monitoring of the stock market data stream is one of the challenging use cases of stream processing systems. By monitoring real-time stock price patterns via high-throughput event stream processing, real-time stock trading advice can be generated. In this paper, we describe our implementation of a system for the DEBS 2022 Grand Challenge to extract breakout patterns from real-time stock market data. Breakout patterns, based on exponential moving averages crossover events, indicate potential bullish or bearish trends, giving investors insight and the opportunity to buy or sell at optimal times. We report details of our high-performance implementation to extract such patterns from a real-time stream to generate stock trading advice event notification outputs. Furthermore, we report the architectural design of our system for parallel processing of data stream, which we implemented from scratch using Python and Java programming languages.","PeriodicalId":263415,"journal":{"name":"Proceedings of the 16th ACM International Conference on Distributed and Event-Based Systems","volume":"9 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"117091305","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Predicate-based push-pull communication for distributed CEP","authors":"Steven Purtzel, Samira Akili, M. Weidlich","doi":"10.1145/3524860.3539640","DOIUrl":"https://doi.org/10.1145/3524860.3539640","url":null,"abstract":"Complex event processing (CEP) enables the detection of situations of interest by evaluating queries over event streams. When applied in a networked application, events generated by distributed nodes are sent over the network to evaluate CEP queries. To reduce the transmission of events, push-based communication that sends each event immediately upon generation may be complemented with a pull-based model that buffers events locally until they are requested for query evaluation. Existing approaches that leverage push-pull communication to reduce transmission costs for distributed CEP, however, exploit solely temporal constraints imposed by a query. As such, they are not applicable in scenarios, in which relevant events may occur in each time window defined by a query. In this paper, we propose predicate-based push-pull (PrePP) plans for CEP queries to overcome the above limitations. Our idea is to construct pull requests that enable fine-granular filtering at event sources based on query predicates, thereby reducing event transmission. Since the construction of optimal PrePP plans is NP-hard, we introduce a set of algorithms to speed up the plan construction by up to five orders of magnitude compared to a brute-force approach, while producing near-optimal results. In extensive experiments, we demonstrate that PrePP plans reduce event transmission by up to three orders of magnitude over baseline techniques.","PeriodicalId":263415,"journal":{"name":"Proceedings of the 16th ACM International Conference on Distributed and Event-Based Systems","volume":"13 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127210247","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Knowledge graph stream processing at the edge","authors":"J. D. Oliveira, Christophe Callé, Weiqin Xu, P. Calvez, Olivier Curé","doi":"10.1145/3524860.3539644","DOIUrl":"https://doi.org/10.1145/3524860.3539644","url":null,"abstract":"We present a knowledge graph management system designed to run on Edge computing devices that handles high-frequency data streams. During the design phase, we took into account the inherent limitations of the devices, i.e., limited computing power and storage space, as well as the expectations of applications, e.g., low latency, high throughput, and intelligent data management. This results in a compact, decompression-free, in-memory, streaming-enabled RDF store that supports continuous querying and some forms of reasoning. The system addresses efficient query processing of data continuously arriving at a fast pace and is well-adapted to event-driven applications such as anomaly and risk detection. We empirically emphasize its accuracy, robustness, latency, and throughput properties on a real-world IoT setting originating from the energy management domain.","PeriodicalId":263415,"journal":{"name":"Proceedings of the 16th ACM International Conference on Distributed and Event-Based Systems","volume":"17 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125070982","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Optimizing complex event forecasting","authors":"V. Stavropoulos, E. Alevizos, Nikos Giatrakos, A. Artikis, Alexander, Artikis","doi":"10.1145/3524860.3539810","DOIUrl":"https://doi.org/10.1145/3524860.3539810","url":null,"abstract":"In Complex Event Recognition (CER), applications express business rules in the form of patterns and deploy them in a CER Engine which seeks the occurrence of such patterns on incoming streams. This is useful for practical applications which rely on the timely detection of patterns to support critical decisions. One step further, stakeholders want to act proactively, accurately forecasting the occurrence of patterns on raw streams well ahead of time to better schedule their decisions. This calls for making the transition from CER to Complex Event Forecasting (CEF). In CEF, stochastic models of future behavior are embedded into the event processing loop to project into the future the sequence of events that have occurred so far and to estimate the likelihood of the imminent occurrence of more complex patterns. CEF performance engages the stochastic model's training speed and forecast accuracy. In turn, these performance dimensions are affected by few parameters. However, CEF parameter tuning so that optimal CEF performance is achieved is a non-trivial task. This is due to the fact that there is an infinite number of possible parameter combinations, each affecting CEF performance in ways which are hard to predict. In this work, we introduce the first CEF Optimizer that gracefully automates CEF parameter tuning decisions, rapidly cherry picking good CEF configurations. We detail the novel internal architecture of our CEF Optimizer and present an elaborate empirical analysis on two applications that illustrates the effectiveness of our optimization approach.","PeriodicalId":263415,"journal":{"name":"Proceedings of the 16th ACM International Conference on Distributed and Event-Based Systems","volume":"17 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114197840","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"PANDA: performance prediction for parallel and dynamic stream processing","authors":"Pratyush Agnihotri, B. Koldehofe, Carsten Binnig, Manisha Luthra","doi":"10.1145/3524860.3543281","DOIUrl":"https://doi.org/10.1145/3524860.3543281","url":null,"abstract":"Distributed Stream Processing (DSP) systems highly rely on parallelism mechanisms to deliver high performance in terms of latency and throughput. Yet the development of such parallel systems altogether comes with numerous challenges. In this paper, we focus on how to select appropriate resources for parallel stream processing under the presence of highly dynamic and unseen workloads. We present PANDA that provides a novel learned approach for highly efficient and parallel DSP systems. The main idea is to provide accurate resource estimates and hence optimal parallelism degree using zero-shot cost models to ensure the performance demands.","PeriodicalId":263415,"journal":{"name":"Proceedings of the 16th ACM International Conference on Distributed and Event-Based Systems","volume":"27 Suppl 1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124572730","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Real-time stock market analytics for improving deployment and accessibility using PySpark and Docker","authors":"Suyeon Wang, Jaekyeong Kim, Yoonsang Yang, Jinseong Hwang, Jungkyu Han, Sejin Chun","doi":"10.1145/3524860.3539652","DOIUrl":"https://doi.org/10.1145/3524860.3539652","url":null,"abstract":"Making timely-decisions amid the massive influx of financial data is one of the essential features of stock market analytics. Many stock market analytics should provide functionalities that compute multiple technical indicators simultaneously and detect breakout situations. The DEBS 2022 Grand Challenge (DEBS22 GC) competition requires to answering two types of queries: technical trend indicators and detection of crossover patterns. In response to the competition, we propose a real-time stock market analytic solution using PySpark and Docker. Our solution calculates the technical trend indicator---Exponential Moving Average(EMA)---in real-time with the window function. With the technical indicators computed, we detect the breakout pattern that helps determine either buy or sell stocks. Our solution not only improves the speed of deploying applications using a Docker container image but also can be accessed easily via a web-based Jupyter notebook.","PeriodicalId":263415,"journal":{"name":"Proceedings of the 16th ACM International Conference on Distributed and Event-Based Systems","volume":"79 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121181194","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"CougaR","authors":"Evangelos Kolyvas, Spyros Voulgaris","doi":"10.1145/3524860.3539805","DOIUrl":"https://doi.org/10.1145/3524860.3539805","url":null,"abstract":"Despite their development for over a decade, a key problem block-chains are still facing is scalability in terms of throughput, typically limited to a few transactions per second. A fundamental factor limiting this metric is the propagation latency of blocks through the underlying peer-to-peer network, which is typically constructed by means of random connectivity. Disseminating blocks fast improves not only the transaction throughput, but also the security of the system as it reduces the probability of forks. In this paper we present CougaR: a simple yet efficient, eclipse-resistant, decentralized protocol that substantially reduces the block dissemination time in blockchain networks. CougaR's key advantages stem from its link selection policy, which combines a network latency criterion with randomness to offer fast and reliable block dissemination to the entire network. Moreover, CougaR is eclipse-resistant by design, as nodes are protected from having all their links directly or indirectly imposed on them by others, which is the typical vulnerability exploited to deploy eclipse attacks. We rigorously evaluate CougaR by an extensive set of experiments, both against a wide spectrum of parameter settings, and in comparison to the current state of the art.","PeriodicalId":263415,"journal":{"name":"Proceedings of the 16th ACM International Conference on Distributed and Event-Based Systems","volume":"47 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116858482","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Materialize","authors":"Frank McSherry","doi":"10.1515/9783034611756.40","DOIUrl":"https://doi.org/10.1515/9783034611756.40","url":null,"abstract":"","PeriodicalId":263415,"journal":{"name":"Proceedings of the 16th ACM International Conference on Distributed and Event-Based Systems","volume":"50 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-06-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121027085","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}