Cognition, oversight, and the safety case for documentation AI.

Abstract

Large language models (LLMs) can now draft hospital discharge summaries. Reported hallucination rates vary by measurement context, from about 1%-2% in structured summarisation to over 60% under provocation testing. The common argument that LLMs outperform a tired resident treats a high-stakes clinical and legal document as a text-quality problem. We argue that a potentially underappreciated concern is the weakening of clinician oversight when reviewing fluent machine-generated text, and that the joint clinician-AI system is central to safety. We propose a Clinician-AI Oversight Model with four safeguards: bounded autonomy, structured verification, error-signal monitoring, and longitudinal evaluation of cognitive effects.