Information theory for complex systems scientists: What, why, and how

Abstract

In the 21st century, many of the crucial scientific and technical issues facing humanity can be understood as problems associated with understanding, modeling, and ultimately controlling complex systems: systems comprised of a large number of non-trivially interacting components whose collective behavior can be difficult to predict. Information theory, a branch of mathematics historically associated with questions about encoding and decoding messages, has emerged as something of a lingua franca for those studying complex systems, far exceeding its original narrow domain of communication systems engineering. In the context of complexity science, information theory provides a set of tools which allow researchers to describe a variety of dependencies, including interactions between the component parts of a system, interactions between a system and its environment, and the mereological interaction between the parts and the “whole”.

In this review aims to provide an accessible introduction to the core of modern information theory, aimed specifically at aspiring (and established) complex systems scientists. This includes standard measures, such as Shannon entropy, relative entropy, and mutual information, before building to more advanced topics, including: information dynamics, measures of statistical complexity, information decomposition, and effective network inference. In addition to detailing the formal definitions, we also make an effort to discuss how information theory can be interpreted and to develop the intuitions behind abstract concepts like “entropy”. The goal is to enable interested readers to understand what information is, and how it is used to better further their own research and education.