TELEIA: A Spanish language dataset for evaluating artificial intelligence models

Abstract

This paper presents TELEIA, a dataset for the evaluation of Spanish language knowledge in Large Language Models (LLMs). TELEIA is designed to complement existing LLMs tests that evaluate many knowledge areas or tasks and are written in English. To evaluate LLMs in Spanish these English tests are translated, which is reasonable for most technical areas and for many tasks, but not when evaluating the knowledge of the Spanish language. New tests specifically designed for Spanish are needed to evaluate the knowledge of the language. This paper introduces TELEIA, a dataset that is an initial step in that direction. The dataset is designed as a set of multiple-choice questions that have the same format and level as those used in several Spanish evaluation tests for humans. The multiple-choice questions enable automation of LLM testing and the use of TELEIA in existing LLM Leaderboards. The questions are divided in three blocks which resemble existing tests of Spanish for foreign learners and for University access. In total, one hundred questions are included that have been prepared and revised by experts on Spanish language, and that have been validated by comparing with the original exams. The dataset will be included in the first Leaderboard of Spanish LLMs.