Development and preliminary testing of a secure large language model-based chatbot for brief alcohol counseling in young adults

Abstract

Objective

Young adults face elevated risks from alcohol use yet encounter significant barriers to accessing evidence-based interventions. Large language models (LLMs) represent a promising advancement for delivering personalized behavioral interventions, but their application to alcohol counseling remains unexplored. This study evaluated the development and preliminary outcomes of a Secure GPT-4-powered text-based Motivational Interviewing Conversational Agent (MICA).

Method

Using a prospective single-arm pilot design, we evaluated MICA across two phases (Phase I: n = 8; Phase II: n = 37), editing the LLM prompts between Phases. Participants aged 18–25 who reported consuming ≥ 10 standard alcohol units weekly completed a counseling session with MICA. We evaluated safety and compared MI fidelity (relational and technical sub-scales of the Client Evaluation of MI [CEMI]) and usability (System Usability Scale) between Phases. We also explored surrogate measures of effectiveness (i.e. proportion of change talk to sustain talk from session logs) and qualitative feedback themes.

Results

No unsafe responses were observed. MI fidelity improved significantly in the CEMI relational sub-scale from Phase I to II (67.2 % to 82.6 %, p = 0.03). Usability remained consistently high across phases (Phase I: 85.4; Phase II: 80.9; p = 0.45). The proportion of within-session change talk was also consistently high (Phase I: 65.2 %; Phase II: 75.8 %; p = 0.10).

Conclusions

This study provides preliminary evidence that LLM-based chatbots can deliver MI-adherent alcohol interventions that are both acceptable to young adults and maintain high MI fidelity. Future research should employ randomized controlled designs with longer follow-up periods to evaluate impact on drinking outcomes.