Run-time adaptation of stream processing spanning the cloud and the edge

Abstract

Applications that process streams of events generated by sensors are important in a wide range of domains. It is now common to distribute stream processing across edge devices and the cloud. This exploits processing power near the sensors, reducing the load on the cloud and often the required network bandwidth. In this paper we focus on one challenge in distributed stream processing: automatically adapting the partitioning of the processing between the edge and the cloud without a loss of service. An example is when the event arrival rate increases and the edge processor can no longer meet performance requirements. Re-partitioning without loss of service involves moving computations between the edge and the cloud while events are still being processed. In this paper we describe StrIoT - a stream processing system that supports automatic re-partitioning. It is based on a set of functional stream operators, and the paper describes how the run-time system can automatically adapt applications that use them. A key feature is support for the fission and fusion of operators during adaptations. Performance evaluation shows that StrIoT can move parts of a stream processing application between the cloud and edge with only a low, temporary impact on performance.