AI-driven report-generation tools in mental healthcare: A review of commercial tools

Artificial intelligence (AI) systems are increasingly being integrated in clinical care, including for AI-powered note-writing. We aimed to develop and apply a scale for assessing mental health electronic health records (EHRs) that use large language models (LLMs) for note-writing, focusing on their features, security, and ethics. The assessment involved analyzing product information and directly querying vendors about their systems. On their websites, the majority of vendors provided comprehensive information on data protection, privacy measures, multi-platform availability, patient access features, software update history, and Meaningful Use compliance. Most products clearly indicated the LLM's capabilities in creating customized reports or functioning as a co-pilot. However, critical information was often absent, including details on LLM training methodologies, the specific LLM used, bias correction techniques, and methods for evaluating the evidence base. The lack of transparency regarding LLM specifics and bias mitigation strategies raises concerns about the ethical implementation and reliability of these systems in clinical practice. While LLM-enhanced EHRs show promise in alleviating the documentation burden for mental health professionals, there is a pressing need for greater transparency and standardization in reporting LLM-related information. We propose recommendations for the future development and implementation of these systems to ensure they meet the highest standards of security, ethics, and clinical care.