Exponential Dissimilarity-Dispersion Family for Domain-Specific Representation Learning

Abstract

This paper presents a new domain-specific representation learning method, exponential dissimilarity-dispersion family (EDDF), a novel distribution family that includes a dissimilarity function and a global dispersion parameter. In generative models, variational autoencoders (VAEs) has a solid theoretical foundation based on variational inference in visual representation learning and are also used as one of core components of other generative models. This paper addresses the issue where conventional VAEs, with the commonly adopted Gaussian settings, tend to experience performance degradation in generative modeling for high-dimensional data. This degradation is often caused by their excessively limited model family. To tackle this problem, we propose EDDF, a new domain-specific method introducing a novel distribution family with a dissimilarity function and a global dispersion parameter. A decoder using this family employs dissimilarity functions for the evidence lower bound (ELBO) reconstruction loss, leveraging domain-specific knowledge to enhance high-dimensional data modeling. We also propose an ELBO optimization method for VAEs with EDDF decoders that implicitly approximates the stochastic gradient of the normalizing constant using log-expected dissimilarity. Empirical evaluations of the generative performance show the effectiveness of our model family and proposed method. Our framework can be integrated into any VAE-based generative models in representation learning. The code and model are available at https://github.com/ganmodokix/eddf-vae