Structured Knowledge Base Enhances Effective Use of Large Language Models for Metadata Curation.

Abstract

Metadata play a crucial role in ensuring the findability, accessibility, interoperability, and reusability of datasets. This paper investigates the potential of large language models (LLMs), specifically GPT-4, to improve adherence to metadata standards in existing datasets. We conducted experiments on 200 random data records describing human samples relating to lung cancer from the NCBI BioSample repository, evaluating GPT-4's ability to suggest edits for adherence to metadata standards. We computed the adherence accuracy of field name-field value pairs through a peer review process, and we observed a marginal average improvement in adherence to the standard data dictionary from 79% to 80% when using GPT-4. We then prompted GPT-4 with domain information in the form of the textual descriptions of CEDAR metadata templates and recorded a statistically significant improvement to 97% from 79% (p<0.01). These results indicate that LLMs show promise for use in automated metadata curation when integrated with a structured knowledge base, though they may struggle when unaided.