A Distributed Topic-Based Pub/Sub Method for Exhaust Data Streams towards Scalable Event-Driven Systems

Abstract

Distributed pub/sub messaging has become indispensable for event-driven systems. There are methods for achieving high scalability regarding topic-based pub/sub by using structured overlay networks. However, these methods waste network resources concerning "exhaust data," which have low or no value most of the time. There are at least two problems: each publisher node continues to forward data to a relay node even if there are no subscribers, and multicast trees are constructed which are excessively large for low value data, namely having a small number of subscribers. In this paper, we formulate the requirements of overlay networks by defining a property called "strong relay-free" as an expansion of relay-free property, and propose a practical method satisfying the property by using Skip Graph. The proposed method involves publishers and subscribers composing connected sub graphs to enable detecting the absence of subscribers and autonomously adjusting the tree size. Through simulation experiments, we confirmed that the proposed method can suspend publishing adaptively, and shorten the path length on multicast trees by more than 75% under an experimental condition with 100,000 nodes. The proposed method is competent for decentralized event-driven systems with encouraging the locally produced data to be consumed locally.