Improving Performance for Geo-Distributed Data Process in Wide-Area

Abstract

Many organizations and end sensors produce massive data around the globe. To analyze the data as a whole, the traditional way is to copy all data to a central datacenter for analysis. This is neither practical nor efficient as the huge transfer data size and the limited network bandwidth. What's more, the data privacy may also matters. Instead of transferring data, we believe moving the computation to where the data is can be a better way to solve this problem. In this paper, we design an algorithm for geo-distributed big data process which is both data-aware and network-aware. Considering the computation's characteristics, we take advantage of data dependency to find out the data locality. And use the integer linear programming (ILP) to achieve network-aware. The implementation of our algorithm is on the top of Spark. We improve the performance of geo-distributed data process by 22% in our experiments.