Insufficient task description can impair in-context learning: A study from information perspective

Abstract

In-context learning, an essential technique in transformer-based models, relies on two main sources of information: in-context examples and task descriptions. While extensive research has focused on the influence of in-context examples, the role of task descriptions remains underexplored, despite its practical significance. This paper investigates how task descriptions impact the in-context learning performance of transformers and how these two sources of information can be effectively fused. We design a synthetic experimental framework to control the information provided in task descriptions and conduct a series of experiments where task description details are systematically varied. Our findings reveal the dual roles of task descriptions: an insufficient task description will cause the model to overlook in-context examples, leading to poor in-context performance; once the amount of information in the task description exceeds a certain threshold, the impact of the task description shifts from negative to positive, and a performance emergence can be observed. We replicate these findings on GPT-4, observing a similar double-sided effect. This study highlights the critical role of task descriptions in in-context learning, offering valuable insights for future applications of transformer models.