Workload distribution with rateless encoding: A low-latency computation offloading method within edge networks

Abstract

In the era of ubiquitous intelligence, user elements offload data-intensive computations to edge network computing clusters, leveraging the efficiency and reliability advantages of distributed computing. However, the delays and failures caused by stragglers significantly hinder system performance. Coded distributed computing combines coding theory with distributed computing, introducing effective redundant computations to accommodate stragglers. Yet, current research often focuses on a fixed number of stragglers with minimal redundancy, lacking a systematic design that considers the inherent heterogeneity in computation, communication, and storage across computing nodes. This paper introduces Rateless Encoding Distributed Computing (REDC), a comprehensive strategy for offloading random arrival computing tasks to distributed computing. REDC devises a rateless coding method for matrix multiplication operations, generating continuous redundant tasks to accommodate random node failures. The proposed queuing theory model requires minimal feedback to update node statuses, dynamically adapting to fluctuations in cluster performance. Simulation results demonstrate that REDC effectively leverages the computing power of clusters with heterogeneous and time-varying characteristics, achieving a resource utilization rate of 93.11%. Moreover, REDC reduces task execution delays by 6.32% compared to the latest baseline, significantly reducing sequential execution delays of computing tasks.