MAVIDSQL: A Model-Agnostic Visualization for Interpretation and Diagnosis of Text-to-SQL Tasks

Abstract

Significant advancements in semantic parsing for text-to-SQL (T2S) tasks have been achieved through the employment of neural network models, such as LSTM, BERT, and T5. The exceptional performance of large language models, such as ChatGPT, has been demonstrated in recent research, even in zero-shot scenarios. However, the inherent transparency of T2S models presents them as black boxes, concealing their inner workings from both developers and users, which complicates the diagnosis of potential error patterns. Despite the fact that numerous visual analysis studies have been conducted in natural language processing communities, scant attention has been paid to addressing the challenges of semantic parsing, specifically in T2S tasks. This limitation hinders the development of effective tools for model optimization and evaluation. This article presents an interactive visual analysis tool, MAVIDSQL, to assist model developers and users in understanding and diagnosing T2S tasks. The system comprises three modules: the model manager, the feature extractor, and the visualization interface, which adopt a model-agnostic approach to diagnose potential errors and infer model decisions by analyzing input–output data, facilitating interactive visual analysis to identify error patterns and assess model performance. Two case studies and interviews with domain experts demonstrate the effectiveness of MAVIDSQL in facilitating the understanding of T2S tasks and identifying potential errors.