Large language models for analyzing open text in global health surveys: why children are not accessing vaccine services in the Democratic Republic of the Congo.

Abstract

Background: This study evaluates the use of large language models (LLMs) to analyze free-text responses from large-scale global health surveys, using data from the Enquête de Couverture Vaccinale (ECV) household coverage surveys from 2020, 2021, 2022 and 2023 as a case study.

Methods: We tested several LLM approaches consisting of zero-shot and few-shot prompting, fine-tuning, and a natural language processing approach using semantic embeddings, to analyze responses on the reasons caregivers did not vaccinate their children.

Results: Performance ranged from 61.5% to 96% based on testing against a curated benchmarking dataset drawn from the ECV surveys, with accuracy improving when LLMs were fine-tuned or provided examples for few-shot learning. We show that even with as few as 20-100 examples, LLMs can achieve high accuracy in categorizing free-text responses.

Conclusions: This approach offers significant opportunities for reanalyzing existing datasets and designing surveys with more open-ended questions, providing a scalable, cost-effective solution for global health organizations. Despite challenges with closed-source models and computational costs, the study underscores LLMs' potential to enhance data analysis and inform global health policy.