RAT: Residual Attention Transformer for Tabular Data

Abstract

The effectiveness of Transformer-based methods in tabular data processing has been extensively validated. However, most existing models primarily adopt Transformer-based encoder architectures, which overly emphasize feature correlations while neglecting the unique explicit information representation characteristics of tabular data. The significantly higher information density of tabular data compared to textual data introduces noise that degrades performance. Consequently, directly applying language models to tabular data often yields suboptimal results. To address this limitation, we propose the Residual Attention Transformer (RAT), a Transformer-based mechanism specifically designed for tabular data. In subsequent sections, the Residual Attention Transformer will be referred to as RAT. RAT introduces residual connections into the self-attention mechanism, effectively preserving raw information while capturing dependencies with higher-order features, thus enhancing the richness of information representation. Additionally, we design a simplified Transformer output head, termed Light-Transformer, which consists of a lightweight Transformer block specifically dedicated to the output stage. This design not only significantly reduces the model’s parameter count, but also greatly improves training and inference efficiency. Extensive experiments were conducted on six public datasets to evaluate our model. The results show that the RAT model consistently outperforms other models in various tasks, confirming its effectiveness and superiority.