NeuralLoss: A Learnable Pretrained Surrogate Loss for Learning to Rank

Abstract

Learning to Rank (LTR) aims to develop a ranking model from supervised data to rank a set of items using machine learning techniques. However, since the losses and ranking metrics involved in LTR are both based on ranking, they are neither continuous nor differentiable, making it challenging to optimize them using gradient descent algorithms. Various surrogate losses have been proposed to address this issue, yet their connection with ranking metrics is often loose, leading to inconsistencies between optimization objectives and evaluation metrics. In this study, we introduce NeuralLoss, a learnable and pretrained surrogate loss. By undergoing training on data structured around ranking metrics, NeuralLoss approximates these ranking metrics, aligning its optimization objectives with evaluation metrics. We employ Transformer to construct the surrogate model and ensure permutation invariance. The pretrained surrogate loss facilitates end-to-end training of ranking models using gradient descent algorithms and can approximate various ranking metrics by adjusting the training data. In this paper, we employ NeuralLoss to approximate NDCG and Recall, demonstrating its performance in both document retrieval and cross-modal retrieval tasks. Experimental results indicate that our approach achieves excellent performance and exhibits strong competitiveness across these tasks.