Towards a foundation model for behavioral intention prediction through multi-source data fusion

In recent years, some studies have attempted to build foundation models to predict user behavioral intentions across various scenarios. Existing methods typically involve two steps: first, using Pre-trained Language Models (PLMs) to convert user historical behaviors into semantic vectors; second, capturing latent user intentions through a behavioral sequence encoder. However, limitations still exist in this pipeline. Firstly, most studies directly fix the parameters of PLMs, making it difficult to optimize the generated semantic vectors for specific application scenarios. Even with fine-tuning PLMs considered, the huge model size brings computational burdens. To address this challenge, we introduce a Low-Rank Adaptation technique, which enhances the quality of semantic vectors by training two lightweight low-rank matrices while keeping the original parameters of PLMs frozen. Additionally, the semantic vectors generated by PLMs often exhibit anisotropy, which weakens the expressive power. To address this, we design a parameter whitening expert and a mixed observation expert module to achieve more isotropic semantic representations. Finally, there are significant differences in user behavior patterns across different domains, and such behavioral conflicts can limit the model’s generalization ability. Therefore, during the pre-training phase, we introduce multi-domain negative sample fusion and design two contrastive learning tasks. By employing a multi-task learning strategy, we enhance the model’s ability to handle multi-domain data. With these improvements, our model achieves stronger adaptability and accuracy in predicting user behavior across different domains.