Robust, Scalable Anomaly Detection for Large Collections of Images

Abstract

A novel robust anomaly detection algorithm is applied to an image dataset using Apache Pig, Jython and GNU Octave. Each image in the set is transformed into a feature vector that represents color, edges, and texture numerically. Data is streamed using Pig through standard and user defined GNU Octave functions for feature transformation. Once the image set is transformed into the feature space, the dataset matrix (where the rows are distinct images, and the columns are features) is input into an original anomaly detection algorithm written by the author. This unsupervised outlier detection method scores outliers in linear time. The method is linear in the number of outliers but still suffers from the curse of dimensionality (in the feature space). The top scoring images are considered anomalies. Two experiments are conducted. The first experiment tests if top scoring images coincide with images which are marked as outliers in a prior image selection step. The second examines the scalability of the implementation in Pig using a larger data set. The results are analyzed quantitatively and qualitatively.