A Dirichlet-Multinomial mixture model of Statistical Science: Mapping the shift of a paradigm

Abstract

Using Bayesian natural language processing (NLP) methods and a scalable variational algorithm tailored for mixtures of discrete positive data, we analyzed a large corpus of 111,411 eprints submitted to the arXiv repository between 1994 and 2022 in the Statistics category (the primary classification for these eprints on arXiv). Our objective is to assess the impact of Machine Learning (ML) on the field of Statistics–specifically, to determine whether the introduction of ML has led to a fundamental paradigm shift, transforming traditional statistical problems or creating entirely new ones, or if this perceived revolution is primarily occurring outside the field of Statistics. Our findings suggest that the only significant paradigm shift for Statistics as a scientific discipline remains the Bayesian revolution that began in the early 1990s.