A Generic Anomaly Detection Approach Applied to Mixture-of-unigrams and Maritime Surveillance Data

Abstract

This paper proposes a new generic method to detect anomalies (i.e., statistical outliers) which can be used with a generative topic model. In this paper, we specify this method in the context of the Mixture-of-unigrams model, which is widely used in text mining. It is possible to detect anomalies with a topic model by applying a threshold to the likelihood. However, it is challenging to choose the threshold since the choice needs to consider both the similarities of the topics and the length of documents. This paper describes a new intuitive method to detect anomalies which simply manipulates the output of the trained model. Such an approach is anticipated to have parameters that are more intuitive to define for a given problem. To assess the utility of the proposed approach, we also present a use case involving identifying ships misreporting their ship-type using geo-location data from the Automatic Identification System (AIS) messages. We show that, if we train a model using data for one type of ship, it is possible to identify ships of another type as anomalous.