Proceedings of the 15th ACM International Conference on Distributed and Event-based Systems最新文献_第3页

Scalable analytics of air quality batches with Apache Spark and Apache Sedona 使用Apache Spark和Apache Sedona对空气质量批次进行可扩展分析

Proceedings of the 15th ACM International Conference on Distributed and Event-based Systems Pub Date : 2021-06-28 DOI: 10.1145/3465480.3466931

Rim Moussa

{"title":"Scalable analytics of air quality batches with Apache Spark and Apache Sedona","authors":"Rim Moussa","doi":"10.1145/3465480.3466931","DOIUrl":"https://doi.org/10.1145/3465480.3466931","url":null,"abstract":"According to the American National Institute of Environmental Health Sciences (NIEHS), air pollutants are harmful to the health of humans and other living beings, and cause damage to the climate and to the ecosystem by polluting lakes, streams, and soils. Recent developments in sensor technology, and Internet of Things (IoT) technologies provide an opportunity to use sensor networks to measure air quality, in real time, at a large number of locations. The adoption and deployment of IoT technologies for sensing air quality raises a challenging research agenda related to big data processing, such as, data analysis, scalable architectures, and algorithms for best managing and processing IoT data at different edges in the IoT ecosystem. In response to the DEBS'2021 contest, we design and implement a scalable solution for comparing previous year and current year air quality indexes for German Cities, as well as the calculus of cities' longest streaks of good air quality. Our solution is designed to be scalable. It's based on primo Apache Spark - an open-source unified analytics engine for large-scale data processing, and secundo Apache Sedona for creating spatial indexes, and performing spatial operations over large-scale spatial data.","PeriodicalId":217173,"journal":{"name":"Proceedings of the 15th ACM International Conference on Distributed and Event-based Systems","volume":"66 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130685849","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1

DEBS grand challenge: real-time detection of air quality improvement with Apache Flink DEBS的重大挑战:用Apache Flink实时检测空气质量的改善

Proceedings of the 15th ACM International Conference on Distributed and Event-based Systems Pub Date : 2021-06-28 DOI: 10.1145/3465480.3466930

J. Marić, K. Pripužić, Martina Antonić

{"title":"DEBS grand challenge: real-time detection of air quality improvement with Apache Flink","authors":"J. Marić, K. Pripužić, Martina Antonić","doi":"10.1145/3465480.3466930","DOIUrl":"https://doi.org/10.1145/3465480.3466930","url":null,"abstract":"The topic of the DEBS Grand Challenge 2021 is to develop a solution for detecting areas in which the air quality index (AQI) improved the most when compared to the previous year. The solution must run two given continuous queries in parallel on the incoming sensor data stream which must return the following: 1) a top 50 cities in terms of AQI improvement with their current AQIs and 2) a histogram of the longest streaks of good AQI. The incoming data is accessed through an API which provides streaming sensor measurements in batches. We present our solution based on Apache Flink, a distributed stream processing framework for the cluster. We opted for Flink since its applications can easily be scaled horizontally and vertically by adding computation nodes or increasing available resources, respectively. Flink allows us to divide the given queries into smaller tasks which can be run concurrently on different nodes in order to reduce the overall processing time and thus improve the performance of our solution. In more detail, the following performance intensive tasks are run in parallel on distributed nodes: 1) retrieving measurement batches, 2) assigning a city to each measurement and 3) calculating air quality index per city. We also discuss the main optimizations we have used to improve the performance and present an experimental evaluation of our solution.","PeriodicalId":217173,"journal":{"name":"Proceedings of the 15th ACM International Conference on Distributed and Event-based Systems","volume":"39 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-06-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"117083344","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 3

Thinking in events: from databases to distributed collaboration software 事件思考:从数据库到分布式协作软件

Proceedings of the 15th ACM International Conference on Distributed and Event-based Systems Pub Date : 2021-06-28 DOI: 10.1145/3465480.3467835

Martin Kleppmann

引用次数: 2

Tutorial on graph stream analytics 图形流分析教程

Proceedings of the 15th ACM International Conference on Distributed and Event-based Systems Pub Date : 2021-06-28 DOI: 10.1145/3465480.3468293

A. Benczúr, Ferenc Béres, Domokos M. Kelen, Róbert Pálovics

引用次数: 0

Towards autonomous semantic stream fusion for distributed video streams 面向分布式视频流的自主语义流融合

Proceedings of the 15th ACM International Conference on Distributed and Event-based Systems Pub Date : 2021-06-28 DOI: 10.1145/3465480.3467837

M. Duc, Anh Le-Tuan, M. Hauswirth, Danh Le-Phuoc

引用次数: 5

Solving the 2021 DEBS grand challenge using Apache Flink 使用Apache Flink解决2021年DEBS的重大挑战

Proceedings of the 15th ACM International Conference on Distributed and Event-based Systems Pub Date : 2021-06-28 DOI: 10.1145/3465480.3466929

Mina N. F. Morcos, Baiqing Lyu, S. Kalathur

引用次数: 1

The synergy of complex event processing and tiny machine learning in industrial IoT 工业物联网中复杂事件处理和微型机器学习的协同作用

Proceedings of the 15th ACM International Conference on Distributed and Event-based Systems Pub Date : 2021-05-04 DOI: 10.1145/3465480.3466928

Haoyu Ren, Darko Anicic, T. Runkler

{"title":"The synergy of complex event processing and tiny machine learning in industrial IoT","authors":"Haoyu Ren, Darko Anicic, T. Runkler","doi":"10.1145/3465480.3466928","DOIUrl":"https://doi.org/10.1145/3465480.3466928","url":null,"abstract":"Focusing on comprehensive networking, the Industrial Internet-of-Things (IIoT) facilitates efficiency and robustness in factory operations. Various intelligent sensors play a central role, as they generate a vast amount of real-time data that can provide insights into manufacturing. Complex event processing (CEP) and machine learning (ML) have been developed actively in the last years in IIoT to identify patterns in heterogeneous data streams and fuse raw data into tangible facts. In a traditional compute-centric paradigm, the raw field data are continuously sent to the cloud and processed centrally. As IIoT devices become increasingly pervasive, concerns are raised since transmitting such an amount of data is energy-intensive, vulnerable to be intercepted, and subjected to high latency. Decentralized on-device ML and CEP provide a solution where data is processed primarily on edge devices. Thus communications can be minimized. However, this is no mean feat because most IIoT edge devices are resource-constrained with low power consumption. This paper proposes a framework that exploits ML and CEP's synergy at the edge in distributed sensor networks. By leveraging tiny ML and μCEP, we now shift the computation from the cloud to the resource-constrained IIoT devices and allow users to adapt on-device ML models and CEP reasoning rules flexibly on the fly. Lastly, we demonstrate the proposed solution and show its effectiveness and feasibility using an industrial use case of machine safety monitoring.","PeriodicalId":217173,"journal":{"name":"Proceedings of the 15th ACM International Conference on Distributed and Event-based Systems","volume":"40 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-05-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122720303","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 18