Comparison of Natural Language Processing and Traditional Qualitative Analyses of Caregiver Feedback From a Community-Based Intervention for Children With Autism.

Abstract

Background: Satisfaction data is a commonly used construct to evaluate programs, products, and experiences. This data is collected through forced choice (scales, surveys) and/or open-ended measures. While open-ended questions have the greatest potential for more useful and candid feedback, analysis of this data can be time-consuming.

Objective: This paper compares two evaluation methods, traditional qualitative analysis (TQA) and Natural Language Processing (NLP), to analyze themes from open-ended satisfaction data of a caregiver-mediated intervention for families of autistic children, called 'Mind the Gap (MTG)'.

Methods: This secondary analysis utilized existing survey data from 73 caregivers that participated in the MTG trial. To identify themes, TQA was performed using latent coding, and R statistical software was used for NLP, with strategies including term frequency and sentiment analysis. Methods were evaluated independently and in combination.

Results and conclusion: Both methods are comparable in thoroughly analysing this type of data, with each method contributing both unique and overlapping themes. TQA and NLP showed parent satisfaction with materials and organization, as well as peer coach support. NLP may provide more thorough results while also reducing the time burden associated with TQA, although this strategy may also overlook important nuance in data. This clinical trial is registered as 'Mind the Gap (AIRB-3) (MTG)' at clinicaltrials.gov, registration number NCT03711799.