Using Virtual Events for Edge-based Data Stream Reduction in Distributed Publish/Subscribe Systems

Abstract

Distributed publish/subscribe systems are an enabling technology for Industrial Internet of Things applications. While the number of sensors increases, network bandwidth becomes a bottleneck. Existing solutions typically aim to reduce network load either by pre-processing events directly on the edge or by aggregating events into larger batches. However, these approaches are rather static and do not adequately account for the application requirements of subscribers or the actual values of sensor measurements. This paper introduces methods for publish/subscribe systems to dynamically adapt payloads of events at runtime based on i) different data reduction and transformation strategies, ii) a wrapper solution around existing message brokers and iii) a semantics-based event schema registry. Consumers are able to subscribe to various quality levels and receive virtual events, that are reconstructed directly at the subscriber based on knowledge from the semantic model and dynamic decision rules. Our evaluation shows that the concept of virtual events can reduce the network load between publishers, the message broker and subscribers compared to multiple investigated compression techniques.