Integrative modeling enables ChatGPT to achieve average level of human counselors performance in mental health Q&A

Abstract

Recent advancements in generative artificial intelligence (GenAI), particularly ChatGPT, have demonstrated significant potential in addressing the persistent treatment gap in mental health care. Systematic evaluation of ChatGPT’s capabilities in addressing mental health questions is essential for its large-scale application. The current study introduces a computational evaluation framework centered on perceived information quality (PIQ) to quantitatively assess ChatGPT’s capabilities. Leveraging datasets of question-answer pairs generated by both humans and ChatGPT, the framework integrates predictive modeling, explainable modeling, and prompt-engineering-based validation to identify intrinsic evaluation metrics and enable automated assessments. Results revealed that unprompted ChatGPT’s PIQ is significantly lower than that of human counselors overall, with notable deficiencies such as insufficient conversational length, lower text diversity, and reduced professionalism. Despite not matching the top 25% of human counselors, our evaluation framework improved ChatGPT’s mean PIQ by 8.91% to 11.67% across four risk levels. Prompted ChatGPT performed comparably to human counselors in severe (p = 0.0561) and moderate-risk questions (p = 0.7851), and significantly outperformed them in low- and no-risk categories by 6.80% and 4.63%, respectively (p < 0.001). However, undesirable verbal behaviors still persist in text diversity and professionalism. These findings validate ChatGPT’s capabilities to address mental health questions while cautioning that further researches are necessary for LLM-based mental health systems to deliver services comparable to human experts.