Gossiping GANs: Position paper

Abstract

A recently celebrated kind of deep neural networks is Generative Adversarial Networks. GANs are generators of samples from a distribution that has been learned; they are up to now centrally trained from local data on a single location. We question the performance of training GANs using a spread dataset over a set of distributed machines, using a gossip approach shown to work on standard neural networks [1]. This performance is compared to the federated learning distributed method, that has the drawback of sending model data to a server. We also propose a gossip variant, where GAN components are gossiped independently. Experiments are conducted with Tensorflow with up to 100 emulated machines, on the canonical MNIST dataset. The position of this paper is to provide a first evidence that gossip performances for GAN training are close to the ones of federated learning, while operating in a fully decentralized setup. Second, to highlight that for GANs, the distribution of data on machines is critical (i.e., i.i.d. or not). Third, to illustrate that the gossip variant, despite proposing data diversity to the learning phase, brings only marginal improvements over the classic gossip approach.