Embeddings as Dirichlet counts: Attention is the tip of the iceberg.

Abstract

Despite the overtly discrete nature of language, the use of semantic embedding spaces is pervasive in modern computational linguistics and machine learning for natural language. I argue that this is intelligible if language is viewed as an interface into a general-purpose system of concepts, in which metric spaces capture rich relationships. At the same time, language embeddings can be regarded, at least heuristically, as equivalent to parameters of distributions over word-word relationships.