Towards efficient compression and communication for prototype-based decentralized learning

In prototype-based federated learning, the exchange of model parameters between clients and the master server is replaced by transmission of prototypes or quantized versions of the data samples to the aggregation server. A fully decentralized deployment of prototype-based learning, without a central aggregator of prototypes, is more robust upon network failures and reacts faster to changes in the statistical distribution of the data, suggesting potential advantages and quick adaptation in dynamic learning tasks, e.g., when the data sources are IoT devices or when data is non-iid. In this paper, we face the challenge of designing an efficient prototype-based decentralized learning network by reducing the overheads in communication and computation. This allows enhancing the scalability of the global system, specially for IoT settings with resource-limited devices. First, we compress the prototype size by applying a clustering algorithm. After that, we filter the prototypes to be disseminate using an information-theoretic measure to share only relevant models or models that provide new knowledge to their neighbors. Then, we define a parallel gossip algorithm to disseminate these models within the learning network. Finally, we define a suitable scheduler able to manage the set of prototypes received to optimize the aggregation phase. In order to validate our proposal we present an analysis of the parallel gossip algorithm regarding the age-of-information (AoI). Our experimental results show the communications load can be substantially reduced without decreasing the convergence rate of the learning algorithm.