A Study of Bidirectional Encoder Representations from Transformers for Sequential Recommendations

Sequential recommender systems seek to capture information about user affinities and behaviors considering their sequential series of interactions. While former models based on Markov Chains and Recurrent Neural Networks were used to model future interactions with the last interaction, sequential recommendation aspires to express more than that. In this paper, we detail BERT4Rec, a sequential recommendation approach, based on bidirectional encoder of self-attention based Transformer mechanisms. Such model, inspired by recent advances in NLP, learns to predict a masked item in a user’s sequence using the Cloze objective. The model unmasks the randomly masked items by capturing attention states from both left and right sides and short-term and long-term contexts. Such a sequential model has been applied in what follows to new datasets from Amazon reviews of different characteristics. Thus, our main contribution consists of testing this approach and its limits on several recommendation datasets.