Open-ViTabQA: A novel benchmark for Vietnamese question answering on open domain wikipedia table

Abstract

This paper presents Open-ViTabQA, the first Vietnamese dataset for Table Question Answering (Table QA), addressing the lack of resources for Vietnamese natural language processing. The dataset was meticulously constructed and rigorously validated to ensure high quality. A comprehensive analysis of the structural characteristics of the dataset, including table structure, question types, and answer patterns, is presented. We also introduce BIF, a novel metric combining PhoBERT embeddings within BERTScore for semantic similarity and ViNLI for logical consistency, effectively capturing Vietnamese-specific linguistic nuances and logical coherence. The rigorously validated dataset, accompanied by an analysis of its structural characteristics, provides a robust framework for evaluating Table QA systems. Experiments with pre-trained models and large language models (LLMs) show that ViT5 achieves an F1-score of 45.22 %, an Exact Match (EM) score of 45.13 %, and a BIF score of 0.562. Among large language models, Gemini 2.0 Flash Experimental achieves 60.50 % F1 and 60.20 % EM, while Gemini 1.5 Pro-leads with a BIF score of 0.649, slightly outperforming Gemini 2.0 Flash Experimental (0.644 BIF), indicating more stable reasoning capabilities. However, a significant gap persists compared to human performance (86.49 % F1, 83.43 % EM, 0.781 BIF), highlighting challenges in capturing Vietnamese linguistic subtleties and logical intricacies. These findings underscore opportunities for advancing model performance and addressing data scarcity in Vietnamese Table QA. To facilitate reproducibility and further research, the Open-ViTabQA dataset is publicly accessible for research purposes.