Autoencoder Evaluation and Hyper-Parameter Tuning in an Unsupervised Setting

Abstract

This paper aims to introduce a new methodology for evaluating autoencoder performance and to shorten time spent on heuristic analysis during hyper-parameter tuning. Existing methodologies for evaluating hyper-parameter tuning focus on finding known anomalies in a labeled set or minimizing the average per row reconstruction error as a method of model selection. This paper focuses on anomaly detection in a completely unsupervised setting, where labels are not known during model training or evaluation. This approach uses the approximate Full Width Half Max (FWHM) of the histogram of the per row reconstruction error in conjunction with the average per row reconstruction error and the number of anomalies found to define a new method of model selection that aims to maximize the FWHM while minimizing the average per row reconstruction error. This methodology simplifies and speeds up model evaluation by presenting model results in an intuitive manner and simplifies the heuristic analysis needed to determine the "best" model.