DONN: leveraging heterogeneous outer products for CTR prediction

Abstract

A primary strategy for constructing click-through rate models based on deep learning involves combining a multi-layer perceptron (MLP) with custom networks that can effectively capture the interactions between different features. This is due to the widespread recognition that relying solely on a vanilla MLP network is not effective in acquiring knowledge about multiplicative feature interactions. These custom networks often employ product methods, such as inner, Hadamard, and outer products, to construct dedicated architectures for this purpose. Among these methods, the outer product has shown superiority in capturing feature interactions. However, the resulting quadratic form from the outer product operation limits the conveyance of informative higher-order interactions to the MLP. Efforts to address this limitation have led to models attempting to increase interaction degrees to higher orders. However, utilizing matrix factorization techniques to reduce learning parameters has resulted in information loss and decreased performance. Furthermore, previous studies have constrained the MLP’s potential by providing it with inputs consisting of homogeneous outer products, thus limiting available information diversity. To overcome these challenges, we introduce DONN, a model that leverages a composite-wise bilinear module incorporating factorized bilinear pooling to mitigate information loss and facilitate higher-order interaction development. Additionally, DONN utilizes a feature-wise bilinear module for outer product computations between feature pairs, augmenting the MLP with combined information. By employing heterogeneous outer products, DONN enhances the MLP’s prediction capabilities, enabling the recognition of additional nonlinear interdependencies. Our evaluation on two benchmark datasets demonstrates that DONN surpasses state-of-the-art models in terms of performance.