Analyzing Llama 3-based Approach for Axiom Translation from Ontologies.

Abstract

Ontology development involves a top-down approach where ontology engineers and domain experts collaboratively define and evaluate ontological elements and axioms. Translating ontology axioms into natural language can significantly aid in ontology evaluation by making the content more understandable to subject matter experts who may lack a background in knowledge engineering. In this preliminary study, we investigate the potential of large language models (LLMs) in axiom translation from ontologies to facilitate ontology evaluation. We utilize Llama 3 to translate 1,192 ontology axioms across 19 distinct axiom types from five published ontologies. Results show that 163 (13.67%) of the Llama 3 translation of the axiom are accurately represented, 268 (22.48%) are not accurately represented, and 761 (63.84%) are partially accurate. Our manual evaluation of the Llama 3 translation indicates some competency in producing hierarchical natural language equivalents while revealing some limitations when translating complex axioms. Nonetheless, there are opportunities to improve the results with few-shot training or using LLMs to provide support in knowledge engineering for ontologies.