Research on the product recommendation algorithm based on PySpark and Jupyter notebook

Due to the problems caused by the development of the Internet, such as information redundancy and junk information overflow, it is crucial that e-commercial companies utilize recommendation algorithms to personalize their online shopping system for every user in order to promote sales. After discussing the pros and cons of demographic filtering, content-based filtering and collaborative filtering, the authors mainly focused on collaborative filtering. This article elaborated on how to design the collaborative filtering algorithm and improve its efficiency. Compared to other recommendation algorithms, collaborative filtering can help customers discover potential interests. Moreover, the system only needs feedback matrixes to train the matrix decomposition model and requires no additional relevant features. One major defect of collaborative filtering is called cold start, which means if a new item is added during training, the system cannot create embedding and make a prediction for it. The technology called WALS projection can solve this problem to some degree.