Human-in-the-Loop Suicide Risk Detection and Clinical Escalation in a Generative AI Mental Health Chatbot: Prospective Implementation Study.

Abstract

Background: Mental illness is a leading cause of disability worldwide, affecting more than 1 billion people, and suicide accounts for 720,000-800,000 deaths annually. Although AI-native mental health chatbots have the potential to help address the substantial gap between treatment needs and workforce capacity, their rapid proliferation has outpaced the development of scalable procedures for identifying and responding to suicidal ideation. Although recommendations call for human oversight to support safety monitoring and intervention in AI-based care, research on human clinical oversight and crisis-response procedures remains limited.

Objective: This study aims to implement a human-in-the-loop (HITL) protocol for suicide risk screening, detection, and clinical escalation in a generative AI (GenAI) mental health chatbot.

Methods: In this implementation study of a HITL suicide risk detection and escalation protocol, adults were recruited online to participate in a single-arm trial of an AI-native mental health chatbot. Participants completed baseline screening using the 9-item Patient Health Questionnaire (PHQ-9). In phase 1, participants who endorsed suicidal ideation were referred to a human clinician for further assessment, safety planning, and provision of referral resources, as appropriate. In phase 2, escalation was triggered by either PHQ-9 responses or the detection of spontaneous, implicit, or explicit disclosures of suicidality during chatbot interactions. In both phases, the Ask Suicide-Screening Questions guided clinician-led assessment and risk stratification. Outcomes included the prevalence of suicidal ideation, the rate of successful clinician outreach, the time from risk detection to initiation of clinician outreach, and the results of risk stratification.

Results: In phase 1, 624 participants completed suicide risk screening, of whom 81 (13%) endorsed PHQ-9 item 9 (score ≥1). Clinician outreach was initiated, on average, 11.2 minutes (SD 13.2 minutes; range 1-54 minutes) after the alert, and 25 of the 81 (31%) participants were successfully reached. In phase 2, 650 participants completed screening, of whom 133 (20.5%) screened positive at baseline. During the 30-day postbaseline observation period, 284 risk-alert events were generated among 172 (26.5%) participants, including 50 conversational suicidal ideation alerts. Clinicians successfully reached 73 of the 172 (42.4%) participants. The mean time from risk detection to initiation of clinician outreach was 11.5 minutes (SD 11.7 minutes; range 1-58 minutes).

Conclusions: Proactive, universal suicide risk screening and conversational detection using a HITL protocol can be operationally implemented in a GenAI mental health chatbot. A subset of participants who were successfully reached by a human clinician described the approach as acceptable and helpful. Given that clinical contact and assessment were not completed for a substantial proportion of alerted participants, further research is needed to optimize assessment and follow-up strategies for individuals who cannot be reached and to characterize the impact of these procedures on clinical outcomes. Nevertheless, the safety procedures described herein provide a framework for developing and refining HITL safety responses and protocols for purpose-built, AI-native mental health chatbots.