Tax-Based Mechanisms for Resource Scaling-Out of Stream Big Data Analytics

Abstract

Cloud-based big data platforms provide physical resources for a variety of applications to analyze all forms of data. For the stream big data analytics, a participated task always needs to scale out resources when its input data increases steeply. Typically, the resource scaling out can be achieved by increasing the parallelism degree of the platform based on the experience. However, the resource scaling-out of each task produces additional cost not only from itself but also from other competitive tasks, which brings about great challenges to ensure the efficient utilization of resources. To solve this problem systematically, we consider the resource scaling-out problem as a non-cooperative game and formulate a total cost model including a risk function and a task execution time function. The total cost of resource scaling-out reflects the influence of topology structure for the benefit of a participated task. Hence, two economic classic tax-based incentive policies: Pivotal Mechanism and Externality Mechanism are applied, to stimulate the participation of tasks. We make simulations in different scenarios including node degree and different characteristics of tasks. The simulations results show that our resource scaling-out mechanism can achieve a better performance close to social optimality.