Multi-Scale Transformers with dual attention and adaptive masking for sequential recommendation

Abstract

Sequential recommendation focuses on modeling and predicting a user’s next actions based on their sequential behavior patterns, using the temporal order and dynamics of user actions to provide more personalized and contextual suggestions. Sequential recommendation models rely on limited temporal scales, making it challenging to explicitly capture diverse user behaviors spanning multiple scales. Motivated by this challenge, this paper introduces ScaleRec, an advanced Multi-Scale Transformer architecture augmented with dual attention mechanisms and adaptive masking for sequential recommendation. ScaleRec integrates interaction granularity and context through multi-scale division, segmenting user behavior sequences into patches of varying lengths. Dual attention explicitly models fine-grained interests and coarse-grained preferences, including intra-patch cross-attention and inter-patch self-attention. Specifically, intra-patch cross-attention employs a learnable Gaussian kernel to introduce locality-based inductive biases, capturing fine-grained behavioral dynamics. The inter-patch self-attention is further enhanced by a Context-adaptive Preferences Aggregator, which dynamically selects and integrates relevant long-term user preferences. Additionally, we introduce an adaptive masking fusion strategy to filter redundant information dynamically. Extensive experiments on six benchmark datasets show that ScaleRec achieves state-of-the-art performance, improving the recommendation performance by up to 24.95% in terms of HR@5. The code of the proposed model is available at: https://github.com/gangtann/ScaleRec.