Skewness-Aware Stream Partitioning: A Key Splitting Suppression Method for Distributed Stream Processing

Abstract

Partitioning algorithms are crucial in distributed stream computing, as they directly affect load balancing across downstream task instances, cluster performance utilization, and data stream processing efficiency. However, existing algorithms suffer from key limitations: they fail to effectively balance load imbalance and key-splitting overhead as excessive splitting increases aggregation costs and memory usage, while insufficient splitting fails to reduce skew. In addition, they cannot timely identify newly emerging high-frequency keys, because historical frequency records dominate detection, which worsens load imbalance over time. To address these limitations, we propose Sa-Stream, a skewness-aware partitioning framework that rapidly captures changes in stream skewness and dynamically adjusts partitioning decisions, thereby balancing load imbalance and key-splitting overhead under evolving stream distributions. (1) A key frequency decay mechanism progressively reduces the influence of historical frequency statistics to improve the prediction of evolving key frequencies. (2) An adaptive threshold modulation method dynamically adjusts the hot-key threshold based on the current key frequency distribution, balancing load imbalance and aggregation costs. (3) A multi-tiered key segregation mechanism classifies keys into cold, warm, and hot categories: cold keys use deterministic hash routing to minimize aggregation costs, hot keys use degree-adjusted key splitting to reduce load imbalance, and warm keys use one-time key splitting to bridge the two extremes. Experiments conducted on Apache Storm demonstrate that Sa-Stream outperforms existing stream partitioning approaches, increasing throughput by up to 16% and reducing latency by up to 50%. Additionally, key splitting is reduced by up to 18%, and load imbalance is decreased by up to 98%.