Word Embeddings Improvement via Echo State Networks

Abstract

The paper continues investigations on the application of bidirectional echo state networks (BiESN) to the task of word sense disambiguation (WSD). Motivated by observations that the quality of the embedding vectors used to train the models influences to a significant degree their accuracy, here we propose the application of a single ESN reservoir to generate new potentially better embedding vectors with different dimensions. BiESN models for WSD of various reservoir sizes were trained using various combinations of new and original embeddings models for the input and/or output steps; the achieved accuracy is reported here. The results demonstrate increased WSD accuracy in several cases of newly derived embedding sets.