Scalable Task Deployment System Inspired from Virus Propagation Models for Large Distributed Workflow Based Systems

Abstract

Deploying, executing and managing large task based workflows on cloud or distributed systems can be challenging. This paper proposes a solution for deploying and launching task based applications on large scale distributed systems. Scaling the existing systems to hundreds of thousands or million of nodes add significant overhead and in some cases will slow down the scalable systems below Amdahl's law because of the cost of managing so many nodes. The proposed solution for task deployment is inspired from computer virus models and realizes the deployment in an exponential fashion by starting with a single container that self divides until it populates all nodes in the datacenter. The architectural model is having the structural shape of binary tree, task metadata messages are routed on the tree model, longest message travel distance is at most log2(n). We compare two solutions for large task deployment and execution. A classical solution of using a linear method for deployment is compared with the computer virus inspired propagation model. Experimental results confirm that this solution is suitable for task based applications that can scale to few millions of worker nodes. Sending messages from the master node to worker nodes should not be a problem according to our simulation.