Can AI replace clinician-rated depression scales? The psychometric properties of HAMLET – Hamilton Large-Language-Model Evaluation Tool

Abstract

Background

The Hamilton Depression Rating Scale (HAMD) is the gold standard for assessing depression but requires clinician administration, limiting its accessibility. Large language models (LLMs) offer potential for automated, valid assessments. We developed HAMLET (Hamilton Large-language-model Evaluation Tool), an interactive LLM-based tool designed to replicate the HAMD-17, and compared its results with those obtained by a psychiatrist.

Methods

HAMLET utilizes Qwen-Max in the temperature of 0.7 and a top-p value of 0.6, guided by structured prompts engineered via clinician-supervised tuning. 60 patients with Major Depressive Disorder completed: (1) HAMLET, (2) clinician-rated HAMD-17, and (3) PHQ-9. Agreement was assessed using Intraclass Correlation Coefficient (ICC), Bland-Altman plots, and Gwet’s AC2. Correlations with HAMD and incremental validity beyond the PHQ-9 were evaluated.

Results

HAMLET demonstrated strong overall agreement with clinician-rated HAMD scores (ICC=0.911; 95 % CI: 0.855–0.946). It outperformed the PHQ-9 in correlating with HAMD scores (r = 0.92 vs 0.79; Steiger’s Z = 3.798, p < 0.001) and demonstrated incremental validity (ΔR²=0.252). Item-level agreement (Gwet’s AC2) was > 0.60 for all items, though was lower for sensitive questions.

Conclusion

HAMLET is the first LLM framework to autonomously conduct HAMD-17 assessments demonstrating substantial agreement with clinician ratings. It combines the rigor of clinician-rated scales with the accessibility of self-report tools. This demonstrates the feasibility of LLMs for scalable, low-cost psychiatric assessment. Future work should address contextual limitations and explore multimodal integration.