Dynamic algorithm selection for the logic of tasks in IoT stream processing systems

Abstract

Various Internet of Things (IoT) and Industry 4.0 use cases, such as city-wide monitoring or machine control, require low-latency distributed processing of continuous data streams. This fact has boosted research on making Stream Processing Frameworks (SPFs) IoT-ready, meaning that their cloud and IoT service management mechanisms (e.g., task placement, load balancing, algorithm selection) need to consider new requirements, e.g., ultra low latency due to physical interactions. The algorithm selection problem refers to selecting dynamically which internal logic a deployed streaming task should use in case of various alternatives, but it is not sufficiently supported in current SPFs. To the best of our knowledge, this work is the first to add this capability to SPFs. Our solution is based on i) architectural extensions of typical SPF middleware, ii) a new schema for characterizing algorithmic performance in the targeted context, and iii) a streaming-specific optimization problem formulation. We implemented our solution as an extension to Apache Storm and demonstrate how it can reduce stream processing latency by up to a factor of 2.9 in the tested scenarios.