Learning deep generative models based on binomial log-likelihood

Abstract

Likelihood-based learning algorithms for deep generative models mostly use the Gaussian log-likelihood. One notable exception is the binomial log-likelihood used in the Wasserstein autoencoder; however, it is not commonly used in practice because it does not generalize well. In this paper, we reconsider the binomial log-likelihood for learning deep generative models and study its theoretical properties. We propose two modifications to the original binomial log-likelihood and derive the convergence rates of the corresponding maximum likelihood estimators. These theoretical results explain why the original binomial log-likelihood performs poorly. In addition, motivated by the modified binomial log-likelihood, we propose a parametric heterogeneous Gaussian log-likelihood, which is novel in learning deep generative models. By analyzing various benchmark image datasets, we show that the proposed parametric heterogeneous Gaussian log-likelihood outperforms the standard homogeneous Gaussian log-likelihood. Additionally, we provide several pieces of evidence to explain why the proposed heterogeneous Gaussian log-likelihood works better than others.