Minimally informed linear discriminant analysis: Training an LDA model with unlabelled data

Abstract

Linear Discriminant Analysis (LDA) is one of the oldest and most popular linear methods for supervised classification problems. Computing the optimal LDA projection vector requires calculating the average and covariance of the feature vectors of each class individually, which necessitates class labels to estimate these statistics from the data. In this paper we demonstrate that, if some minor prior information is available, it is possible to compute the exact projection vector from LDA models based on unlabelled data. More precisely, we show that either one of the following three pieces of information is sufficient to compute the LDA projection vector if only unlabelled data are available: (1) the class average of one of the two classes, (2) the difference between both class averages (up to a scaling), or (3) the class covariance matrices (up to a scaling). These theoretical results are validated in numerical experiments, demonstrating that this minimally informed Linear Discriminant Analysis (MILDA) model closely approximates the solution of a supervised LDA model, even on high-dimensional, poorly separated or extremely imbalanced data. Furthermore, we show that the MILDA projection vector can be computed in a closed form with a computational cost comparable to LDA and is able to quickly adapt to non-stationary data, making it well-suited to use as an adaptive classifier that is continuously retrained on (unlabelled) streaming data.