Efficient Solvers for Wyner Common Information With Application to Multi-Modal Clustering

Abstract

In this work, we propose computationally efficient solvers for novel extensions of Wyner common information. By separating information sources into bipartite, the proposed Bipartite common information framework has difference-of-convex structure for efficient non-convex optimization. In known joint distribution cases, our difference-of-convex algorithm(DCA)-based solver has a provable convergence guarantee to local stationary points. As for unknown distribution settings, the insights from DCA combined with the exponential family of distributions for parameterization allows for closed-form expressions for efficient estimation. Furthermore, we show that the Bipartite common information applies to multi-modal clustering without employing ad-hoc clustering algorithms. Empirically, our solvers outperform state-of-the-art methods in clustering accuracy and running time over a range of non-trivial multi-modal clustering datasets with different number of data modalities.