Beyond decomposition: Hierarchical dependency management in multi-document question answering

Abstract

When using retrieval-augmented generation (RAG) to handle multi-document question answering (MDQA) tasks, it is beneficial to decompose complex queries into multiple simpler ones to enhance retrieval results. However, previous strategies always employ a one-shot approach of question decomposition, overlooking subquestions dependency problem and failing to ensure that the derived subqueries are single-hop. To overcome this challenge, we introduce a novel framework called DSRC-QCS. Decompose-solve-renewal-cycle (DSRC) is an iterative multi-hop question processing module. The key idea of DSRC involves using a unique symbol to achieve hierarchical dependency management and employing a cyclical process of question decomposition, solving, and renewal to continuously generate and resolve all single-hop subquestions. Query-chain selector (QCS) functions as a voting mechanism that effectively utilizes the reasoning process of DSRC to assess and select solutions. We compare DSRC-QCS against five RAG approaches across three datasets and three LLMs. DSRC-QCS demonstrates superior performance. Compared to the Direct Retrieval method, DSRC-QCS improves the average F1 score by 17.36% with Alpaca-7b, 10.83% with LLaMa2-Chat-7b, and 11.88% with GPT-3.5-Turbo. We also conduct ablation studies to validate the performance of both DSRC and QCS and explore factors influencing the effectiveness of DSRC. We have included all prompts in the Appendix.