Do Hebbian synapses estimate entropy?

Abstract

Hebbian learning is one of the mainstays of biologically inspired neural processing. Hebb's (1949) rule is biologically plausible, and it has been extensively utilized in both computational neuroscience and in unsupervised training of neural systems. In these fields, Hebbian learning became synonymous for correlation learning. But it is known that correlation is a second order statistic of the data, so it is sub-optimal when the goal is to extract as much information as possible from the sensory data stream. We demonstrate how information learning can be implemented using Hebb's rule. Thus the paper brings a new understanding to how neural systems could, through Hebb's rule, extract information theoretic quantities rather than merely correlation.