Design and Implementation of Smart City Big Data Processing Platform Based on Distributed Architecture

Abstract

For the problem about the low shared efficiency and the poor integrating degree of the massive multivariate data, in the paper, we propose a design scheme based on distributed architecture, mainly using Kafka, Storm and Spark clusters, and the real-time transmission among the structured data, unstructured data and database data are implemented. And the real-time analysis system of real-time data and statistical analysis system of offline data are developed. Finally, the test results show the performance of Big Data processing platform, and demonstrate fully the advantage of the distributed architecture.