Entropy-Guided Meta-Initialization regularization for few-shot text classification

Abstract

Meta-learning has emerged as the dominant approach for tackling the problem of limited datasets in text classification and achieved state-of-the-art performance. In particular, gradient-based meta-learning finds good meta-initialization (parameters) through learning on a variety of tasks, which is known to be effective for the few-shot problem. However, we find that meta-initialization of the existing method has an over-confidence for the specific class before task adaptation, which potentially leads to deterioration in generalization performance. To address this issue, we propose a simple and effective Entropy-Guided Meta-Initialization regularization (EGMI) method. The proposed EGMI focuses on maximizing entropy so that the model has meta-initialization to prevent over-confidence for specific classes. In our experiments, we show that EGMI outperforms the current state-of-the-art methods on few-shot benchmark datasets by a large margin. In particular, we achieve a performance improvement from 84.13% to 91.36% on the 15-way 5-shot of the clinc150 dataset. We also do not need any additional parameters and the training cost does not increase. Our code is publicly available https://anonymous.4open.science/r/EGMI-9D60/README.md.