Feature Drift Detection using Overlapping Window and Mann-Whitney U Test

Abstract

As data is ubiquitous in several real-world problems, data stream mining is a rapidly growing research area. It is expected that data stream sources will undergo changes in data distribution due to their ephemeral nature, which is called concept drift. There has been a very scant study of one particular type of drift, namely feature drift, so this paper aims to explore that type of drift. As a result of feature drift, learners must detect and adapt to changes in the relevant subset of features and the changing nature of the learning task itself. An approach to detecting feature drift was developed in this work. We used overlapping landmark windowing to keep the previous data's properties windows while analyzing the most recent data. Using the Mann-Whitney U test, we compare and store the distribution of each feature in two consecutive windows. Whenever the statistical properties of the window exclude a particular boundary from the distribution, drift is detected. We validated the effectiveness of our proposal by conducting experiments on real data.