Prompting Is All You Need - Until It Isn't: Exploring the Limits of LLMs for Negation Detection in German Clinical Text.

Abstract

Introduction: Detecting negations in clinical text is crucial for accurate documentation and decision-making.

Methods: This study assesses open-source Large Language Models (LLMs) for detecting negations in German clinical discharge letters, comparing them to the rule-based approach (GeNeg) and human annotations.

Results: While Llama 3.3 and Deepseek-R1 (70B) showed slight accuracy improvements, their high computational costs limit practicality compared to GeNeg. Llama 3.3 achieved the highest accuracy (.9670) and F1-score (.9620), outperforming all other models and slightly exceeding GeNeg in accuracy and F1-score. However, it required significantly more computational time (5.9 sec/sent) when compared to GeNeg's processing time (.005 sec/sent).

Conclusion: The study results suggest hybrid approaches combining rule-based efficiency paired with LLMs' linguistic capabilities. In addition, future work should therefore optimize prompts and integrate LLMs with traditional methods to balance accuracy and efficiency.