Knowledge-Centered Dual-Process Reasoning for Math Word Problems With Large Language Models

Abstract

Math word problem (MWP) serves as a critical milestone for assessing the text mining ability and knowledge mastery level of models. Recent advancements have witnessed large language models (LLMs) showcasing remarkable performance on MWP. However, current LLMs still frequently exhibit logical errors, which highlights their inability to fully grasp the knowledge required for genuine step-by-step mathematical reasoning. To this end, in this paper, we propose a novel Knowledge-guided Solver (KNOS) framework that empowers LLMs to simulate human mathematical reasoning, whose core idea is to Invoke-Verify-Inject necessary knowledge to solve MWP. We draw inspiration from the dual-process theory to construct two cooperative systems: a Knowledge System and an Inference System. Specifically, the Knowledge System employs LLMs as the knowledge base and develops a novel knowledge invoker that can elicit their relevant knowledge to support the strict step-level mathematical reasoning. In the Inference System, we propose a knowledge verifier and a knowledge injector to evaluate the knowledge rationality and further guide the step-wise symbolic deduction in an interpretable manner based on human cognitive mechanism, respectively. Moreover, to tackle the potential scarcity issue of mathematics-specific knowledge in LLMs, we consider an open-book exam scenario and propose an improved version of KNOS called EKNOS. In EKNOS, we meticulously design knowledge selectors to extract the most relevant commonsense and math formulas from external knowledge sources for each reasoning step. This knowledge is utilized to assist the knowledge invoker in better stimulating LLMs’ reasoning abilities. Both KNOS and EKNOS are flexible to empower different LLMs. Our experiments with GPT3, ChatGPT, and GPT4 not only demonstrate their reasoning accuracy improvement but also show how they bring the strict step-wise interpretability of mathematical thinking.