Mining robust neighborhoods for quality control of sensor data

Abstract

Neighborhoods, as used for spatial and spatial-temporal data mining, define areas of similarity in data. Unless defined to account for outliers, missing data and spatial-temporal variation, the robustness of methods utilizing neighborhoods will suffer. The focus of this paper is to demonstrate that neighborhoods can be defined and used in a robust manner that is resistant to such challenges. Our approach employs robust methods in both neighborhood construction and neighborhood application to estimate observations. These methods were tested with a large weather sensor data set from the National Weather Service that includes quality control indicators. Results were compared to a popular method used in the weather community, evaluated by root-mean-squared error and grouped by quality control indicator. Our first time published results show that our methods are robust in the presence of outliers, missing data and spatial-temporal variation, yielding results consistent with quality control labels assigned to the data by the provider by way of an extensive rule-based system, indicating that our approaches show promise for use in quality control assessment.