Scaling Erlang distribution: going beyond the fully connected mesh

Abstract

Distributed Erlang, the process of transparently running Erlang programs over networks, has a long history of immense usefulness but has problems when distributed systems reach certain scales. We explain the issues and show research done towards the goal of transparently enhancing Erlang distribution, so that changes to existing applications and systems can be avoided. We propose several research directions together with prototype implementations that all serve the purpose of improving the current status quo. This includes using different transport protocols, generalizing implementation efforts and incorporating routing protocols for more dynamic node constellations. We then describe some background and history of various work to solve Erlang distribution scalability issues. We show that there is much room for improvement on the Erlang distribution layer without breaking abstractions that developers are used to and rely on.