A Prototype for Analyzing the Internet Routing System Based on Spark and Docker

Abstract

Now, network security is becoming more and more serious, and network security has become a big data issue. But the big data processing technology is complex with low utilization, poor availability and impossible load balancing. In this paper, we propose a highly elastic and available big data processing prototype based on spark and docker. In our prototype, we use microservice as basic unit to provide service for business and docker container as carrier for microservice. By monitoring the actual running state of the business and the lightweight and fast start features of the container, we can quickly and dynamically add and remove containers to provide scaling service for business. Then we download the BGP routing table from www.routeviews.org and use our prototype for analysis. Experiments show that, based on the size of the BGP routing table, our prototype can scale to meet real-time business processing requirements.