Benchmarking In-Context Learning Strategies of Large Language Models for Math Reasoning Tasks

Abstract

The use of large language models (LLMs) in mathematical reasoning has become a cornerstone of related research, demonstrating the intelligence of these models and enabling potential practical applications through their advanced performance, such as in educational settings. Despite the variety of datasets and in-context learning algorithms designed to improve the ability of LLMs to automate mathematical problem solving, the lack of comprehensive benchmarking across different datasets makes it difficult to determine which in-context algorithms are effective, efficient, and suitable for specific educational applications. In this project, we present a benchmark that fairly compares seven state-of-the-art in-context learning algorithms for mathematical problem solving across five widely used mathematical datasets on four powerful foundation models. Beyond accuracy, we explore the tradeoff between efficiency and performance, highlighting the practical applications of LLMs for mathematical reasoning. Our results indicate that larger foundation models, such as GPT-4o and LLaMA 3-70B, can solve mathematical reasoning independently from the concrete prompting strategy, while for smaller models, the in-context learning approach significantly influences the performance. Moreover, the optimal prompt depends on the chosen foundation model. We open source our benchmark code to support the integration of additional models in future research.