On-the-Fly Adaptation of Data Stream Queries

Abstract

Data stream processing can be applied in scenarios where high throughput and real-time response is required. In distributed environments, preprocessing at sensor nodes is beneficial. Processing logic is deployed at sensor nodes and data is processed while streaming through the nodes. However, updating the logic is difficult in such a continuously running environment. Volatile operator states get lost if nodes are restarted for an update. The challenge is to update the operators during the runtime with the preservation of their states. We developed a data stream processing solution which is capable of adapting the logic on-the-fly. States are copied while migrating from the original to new logic. For implementation, we utilized the dynamic component system OSGi. In this paper we present methods for state selection and transfer from the input buffers of the running operators. The state values to be transferred are limited to a minimum quantity. The state transfer methods allow a faster migration compared to other approaches and a guaranteed output, which makes our system real-time compatible.