Investigating legal question generation using large language models

Abstract

We propose the task of legal question generation (QG) as an application in Legal NLP. Specifically, the task is to generate a question, given a context and an optional keyword. We create the first dataset for the QG task in the legal domain, called LegalQ, consisting of 2023 <context, question> pairs spanning the legal systems of multiple countries, and multiple languages. We then use this dataset to benchmark several Large Language Models (LLMs) including Turbo-GPT-3.5, GPT-4, Llama2-70b, Llama2-13b, and Aalap-Mistral-7b (a legal domain-specific LLM). We also fine-tune several open-source LLMs such as T5, BART, Pegasus, and Flan-T5, which helps to improve results over zero-shot prompting of LLMs. We also use the idea of in-context learning (via few-shot examples) to generate questions of varying types and difficulty levels. Furthermore, we introduce a novel domain-specific prompting strategy based on chain-of-thoughts prompting for question generation. Further, we perform Bloom Taxonomy analysis of the questions generated by the LLMs, thereby showing that ‘understanding’ and ‘remembering’ are the two most dominant types of questions generated by the LLMs. Human evaluation of the generated questions shows promise in terms of generating grammatically correct, relevant, appropriate, complex, and novel questions. Finally, we analyze the incomplete or unanswerable generated questions to find possible reasons for these issues.