TsRss: A Practical Stream data Cleaning Method based on Local Shape Feature

Abstract

Stream data which is common usually suffers from dirty data points due to noise interference, unreliable sensor reading, erroneous extraction of stock prices or other various reasons. Existing smoothing filter based data cleaning methods seriously alter the data without preserving the original information. And the others such as SCREEN need to be guided by some semantic constraints in specific application scenarios. To improve the usability, we propose a method called TsRss, which is a practical stream data cleaning method based on local shape feature (Shape-Sheet). TsRss is based on the basic idea that data points failing to match its local shape features are more likely to be dirty. To this end, we first study the methods of generating and representing unequal-length Shape-Sheets based on the local shape features. Then the method for finding dirty data via anomaly detection is proposed based on Shape-Sheet. Finally, experiments were conducted on several real datasets. The result showed that TsRss was more practical in use on various types of data, more accurate or more time-saving compared with state-of-the-art methods.