Using large language models to create lexicons for interpretable text models with high content validity: The suicide risk lexicon.

Abstract

Researchers often want to measure a variety of constructs such as anxiety, discrimination, or loneliness in text data from surveys, interviews, social media, and electronic health records. Using large language models (LLMs)-although optimal for text classification-remain infeasible for some researchers due to concerns around computational expertise, cost, privacy, and compute requirements. Therefore, some researchers prefer to use lightweight models for large data sets or interpretable models to avoid mistakes in high-stakes scenarios. Lexicons offer simple baselines to LLMs by searching for relevant phrases and can be used together with LLMs to guarantee capturing specific keywords. However, building new lexicons is resource intensive. In this study, we found that GPT-4 Turbo was able to automatically create a lexicon for 49 known risk factors for suicidal thoughts and behaviors, which we release as the Suicide Risk Lexicon. Generating a lexicon with LLMs quickly measures most constructs relevant for suicide risk detection, resulting in high content validity. This lexicon was able to accurately predict risk in crisis counseling conversations. After validating the lexicon with clinician ratings, the lexicon modestly outperformed the linguistic inquiry and word count lexicon, which has low content validity for mental illness, and performed similarly to some black-box deep learning models. We discovered that active suicidal ideation and direct self-injury were stronger indicators of imminent risk than passive suicidal ideation and depressed mood in this ecological setting of crisis counseling conversations. To simplify creating new lexicons for other research domains, we introduce a Python package, construct-tracker. (PsycInfo Database Record (c) 2026 APA, all rights reserved).