Using machine learning to automate the collection, transcription, and analysis of verbal-report data.

Abstract

What people think and say during experiments is important for our understanding of the human mind. However, the collection and analysis of verbal-report data in experiments are relatively costly and are thus grossly underutilized. Here, we aim to reduce such costs by providing software that collects, transcribes, and analyzes verbal-report data. Verbal data are collected using jsPsych (De Leeuw, Behavior Research Methods, 47, 1-12, 2015), making it suitable for both online and lab-based experiments. The transcription and analyses rely on classical machine-learning methods as well as deep learning approaches (e.g., large language models), making them substantially more efficient than current methods using human coders. We demonstrate how to use the software we provide in a case study via a simple memory experiment. This collection of software was designed to be modular, allowing for the update and replacement of various components with superior models, as well as the easy addition of new methods. It is our sincere hope that this approach popularizes the collection of verbal-report data in psychology experiments.