Big Data Pipeline Scheduling and Adaptation on the Computing Continuum

Abstract

The Computing Continuum, covering Cloud, Fog, and Edge systems, promises to provide on-demand resource-as-a-service for Internet applications with diverse requirements, ranging from extremely low latency to high-performance processing. However, eminent challenges in automating the resources man-agement of Big Data pipelines across the Computing Continuum remain. The resource management and adaptation for Big Data pipelines across the Computing Continuum require significant research effort, as the current data processing pipelines are dynamic. In contrast, traditional resource management strategies are static, leading to inefficient pipeline scheduling and overly complex process deployment. To address these needs, we propose in this work a scheduling and adaptation approach implemented as a software tool to lower the technological barriers to the management of Big Data pipelines over the Computing Continuum. The approach separates the static scheduling from the run-time execution, em-powering domain experts with little infrastructure and software knowledge to take an active part in the Big Data pipeline adaptation. We conduct a feasibility study using a digital healthcare use case to validate our approach. We illustrate concrete scenarios supported by demonstrating how the scheduling and adaptation tool and its implementation automate the management of the lifecycle of a remote patient monitoring, treatment, and care pipeline.