A short-term memory clustering algorithm for evolving data streams

Data stream clustering is a fundamental problem in many streaming data analysis applications, which faces the following key challenges: (a) efficiently utilizing initial results to update clusters; (b) effectively managing concept drift when dealing with non-stationary data, which leads to decreased clustering accuracy over time. To address these limitations, this paper presents a new short-term memory clustering algorithm for evolving data streams, called STM-Stream. Short-term memory refers to storing the nucleus and radius of cell groups as the window slides, enabling streaming data clustering through three key steps: Firstly, the cell split method is used to obtain the initial data distribution. Then, a novel dynamic projection strategy is used to fuse the stored data distribution with newly arriving data distributions. Finally, based on the updated memory, an adaptive group radius grouping and merging method is designed to produce the final clustering result. Regarding the frequently occurring concept drift issue during clustering, the internal processes of four types of concept drift (Sudden, Gradual, Incremental, and Reoccurring) are analyzed and discussed. The article further extracted two main change processes: Gradual and Sudden drift to characterize the data migration process. Through dynamic projection and adaptive group radius methods, the algorithm can automatically correct its memory with sudden or gradual changes when concept drift occurs, which is equally effective for incremental and occurring concept drift. The experiment demonstrated that STM-Stream can effectively address concept drift, which frequently occurs in continuously generated streaming data, thereby preventing a decline in clustering accuracy over time.