Text generation by probabilistic suffix tree language model

Abstract

During last decade, language modeling has been dominated by neural structures; RNN, LSTM or Transformer. These neural language models provide excellent performance to the detriment of very high computational cost. This work investigates the use of probabilistic language model that requires much less computational cost. In particular, we are interested in variable-order Markov model that can be efficiently implemented on a probabilistic suffix tree (PST) structure. The PST construction is cheap and can be easily scaled to very large dataset. Experimental results show that this model can be used to generated realistic sentences.