Resource Efficiency to Partition Big Streamed Graphs

Abstract

Real time streaming and processing of big graphs is a relevant and challenging application to be executed in a Cloud infrastructure. We have analysed the amount of resources needed to partition large streamed graphs with different distributed architectures. We have improved state of the art limitations proposing a decentralised and scalable model which is more efficient in memory usage, network traffic and number of processing machines. The improvement has been achieved summarising incoming vertices of the graph and accessing to local information of the already partitioned graph. Classical approaches need all information about the previous vertices. In our system, local information is updated in a feedback scheme periodically. Our experimental results show that current architectures cannot process large scale streamed graphs due to memory limitations. We have proved that our architecture reduces the number of needed machines by seven because it accesses to local memory instead of a distributed one. The total memory size has been also reduced. Finally, our model allows to adjust the quality of the partition solution to the desired amount of memory and network traffic.