Distilling Large Language Models for Structured Query Language generation with reverse Kullback–Leibler and dynamic α scheduler

With the increasing reliance on data-driven decision-making, the demand for efficient Structured Query Language (SQL) query generation has grown significantly, as it serves as a crucial bridge between natural language and databases. While Large Language Models (LLMs) excel in this task, their high computational costs and inconsistent effectiveness pose significant limitations. This study introduces a knowledge distillation approach to create efficient, high-performing models for SQL generation. By integrating teacher and student model distributions with a dynamic $\alpha$ scheduler inspired by learning rate schedulers, the method adjusts the teacher’s influence during training, enhancing stability and narrowing performance gaps. Additionally, reverse Kullback–Leibler Divergence (KLD) loss balances contributions, allowing the student model to refine itself while leveraging teacher guidance. The resulting distilled student model, which is 100 times smaller than GPT-4, achieves 80.5% accuracy on benchmark datasets and outperforms GPT-4 in this domain. Furthermore, it demonstrates a 10.2% improvement on extra-hard questions compared to its undistilled counterpart. This work highlights the potential of optimizing LLMs for reduced computational costs and superior performance in SQL query generation and beyond.