Discriminant Analysis Deep Neural Networks

One consensus in the machine learning community is that obtaining good representations of the data is crucial for the classification tasks. But establishing a clear objective for representation learning is an open question and difficult. In this paper, we propose the Discriminant Analysis Loss Function (DALF) for the representation learning in Deep Neural Networks (DNNs). The gradients of DALF explicitly minimize the within-class variances (scatter) and maximize the between-class variances. We use DALF to drive the training of DNNs and call them Discriminant Analysis Deep Neural Networks (DisAnDNNs). Compared to other Linear Discriminant Analysis (LDA)-based cost functions, the computational cost of DALF is drastically reduced by avoiding eigen-decomposition and matrix inversion. We used simple datasets to illustrate the geometric meaning of DALF and compared it with LDA, then experimented with DALF-driven Residual Learning Nets (ResNets) on the pediatric pneumonia (chest X-ray image) dataset. The experimental results show that the DisAnDNNs achieve state-of the-art accuracy in the binary classification task. Particularly, in the pediatric pneumonia dataset, we achieved the accuracy of 96.63%, with a sensitivity of 99.23% and a specificity of 92.30%, all of which are better than the results in the literature that published the dataset.