Can an Algorithm Tell How Spiritual You Are? Using Generative Pretrained Transformers for Sophisticated Forms of Text Analysis.

Abstract

Objective: Text analysis is a form of psychological assessment that involves converting qualitative information (text) into quantitative data. We tested whether automated text analysis using Generative Pre-trained Transformers (GPTs) can match the "gold standard" of manual text analysis, even when assessing a highly nuanced construct like spirituality.

Method: In Study 1, N = 2199 US undergraduates wrote about their goals (N = 6597 texts) and completed self-reports of spirituality and theoretically related constructs (religiousness and mental health). In Study 2, N = 357 community adults wrote short essays (N = 714 texts) and completed trait self-reports, 5 weeks of daily diaries, and behavioral measures of spirituality. Trained research assistants and GPTs then coded the texts for spirituality.

Results: The GPTs performed just as well as human raters. Human- and GPT-generated scores were remarkably consistent and showed equivalent associations with other measures of spirituality and theoretically related constructs.

Conclusions: GPTs can match the gold standard set by human raters, even in sophisticated forms of text analysis, but require a fraction of the time and labor.