A Study on the CBOW Model's Overfitting and Stability

Abstract

Word vectors are distributed representations of word features. Continuous Bag-of-Words Model(CBOW) is a state-of-the-art model for learning word vectors, yet can be ameliorated for learning better word vectors because we find that CBOW is vulnerable to be overfitted and unstable. We use two methods to solve these two problems so that CBOW can learn better word vectors. In this study, we add the regularized structure risk summation to the objective function of the CBOW model and propose inverse word frequency encoding for the CBOW model. Our proposed methods substantially improve the quality of word vectors, boosting r from 0.638 to 0.696 for word relatedness and total accuracy from 30.80% to 38.43% for word pairs relationship relatedness regarding to 52 million training words with 200 dimensionality.