Achieving multilevel elasticity for distributed stream processing systems in the cloud environment: A review and conceptual framework

Abstract

Recent awareness and advances in technology have triggered excessive use of social media, IoT devices, remote sensing devices, mobile applications, web applications, and gaming more than ever before in time. Such platforms are hosting their applications on the cloud as it provides various services on a pay-per-use basis. A Cloud Service Provider (CSP) should deliver all its services very swiftly to process real-time applications on time. Real-time stream computations are characteristically long-lived and receive data in an unpredictable form, requiring a fair amount of resources for their processing in constrained time. Such a dynamic nature of applications demands resource elasticity at runtime. The cloud architecture is stacked with different types of resources, each having a discrete adaption process with distinct elasticity properties. Scaling the absolute amount of resources leads to performance boosting. Recent literature landscapes the elasticity at Virtual Machine (VM) level, describing various techniques for scaling VMs. Each technique targets a distinct aspect with specific assumptions. However, the literature lacks a comprehensive survey at the operator level, where actual processing takes place and has a higher impact on the performance of the system. Compared to other works in the literature, this work presents a detailed analysis of various approaches targeting elasticity at the operator level of cloud architecture for stream processing applications, along with the conceptual framework, scaling at the operator, VM, and server levels. We have also discussed the various elastic approaches for scaling the resources at multilevel: VM and operator-level concurrently, for Distributed Stream Processing (DSP) applications running on the cloud. Conceptually, with the proposed framework, we can attain maximum resource utilization at each layer. In future work, we will evaluate the proposed framework with real-world application.