CMSDPP: Cloud-native Multi-source Streaming Data Processing Platform

Abstract

As the digital transition process of enterprises continues to advance, streaming data processing platforms are becoming the most important part of the enterprise data infrastructure. Meanwhile, cloud-native, an emerging service pattern of cloud computing, gains more and more attention for its ability to reduce the cost of application deployment and maintenance and enhance the effectiveness. In this paper, we design and implement a Cloud-native Multi-source Streaming Data Processing Platform (CMSDPP). It provides various capabilities, such as data access, data aggregation, data analysis and other data processing capabilities, for streaming data processing platform on cloud environment. Besides, we provide a unified registry of data processing components to realize the management of various streaming data processing capabilities. Relying on the extensibility and flexibility of cloud, convenient deployment and management of data application are provided in CMSDPP.