STREAMLINE: Dynamic and Resource-Efficient Auto-Tuning of Stream Processing Data Pipeline Ensembles

Abstract

With the growing volume of data generated by IoT devices and user-driven services, stream processing has become essential for handling continuous, real-time data. However, fluctuating workloads and the dynamic nature of data streams make it difficult to maintain consistent performance over time, requiring adaptive resource allocation and frequent configuration tuning. Running multiple data stream processing pipelines on shared resources further exacerbates the problem by increasing contention, leading to higher end-to-end latency and reduced performance stability. Most existing approaches focus on tuning individual configuration parameters in isolation and overlook interactions between concurrently running data pipelines. To address these limitations, we present STREAMLINE, a dynamic multi-layer auto-tuning framework designed for stream processing environments. STREAMLINE uses transformers to predict future workloads and an evolutionary algorithm to automatically tune configuration parameters. It also includes a resource-efficient scheduler that efficiently assigns operators to resources across a compute cluster. Our dynamic update mechanism minimizes downtime and preserves state during configuration parameter and scheduling changes. We evaluate STREAMLINE on the Grid’5000 testbed using real-time IoT and streaming benchmarks. Results show that STREAMLINE outperforms state-of-the-art methods, improving throughput, end-to-end latency, and CPU utilization by up to 4$\times$ , 10$\times$ , and 9$\times$ , respectively, while reducing costs by up to 10$\times$ .