Development of a novel artificial intelligence clinical decision support tool for hand surgery: HandRAG.

Abstract

Purpose: Hand surgery decision-making requires integration of complex anatomical understanding, diverse patient-specific factors, and nuanced operative techniques. While artificial intelligence (AI), large language models (LLMs), and retrieval-augmented generation (RAG) models have advanced significantly in various fields, no AI-driven clinical decision support systems currently exist for hand surgery. A novel retrieval-enhanced AI large language model specifically tailored for hand surgery was developed, capable of effectively utilizing peer-reviewed published hand surgery literature for clinical decision support in real-time at point of care.

Methods: An AI clinical decision support system was developed integrating all available open-access 4510 peer-reviewed hand surgery publications from 2000 to 2024 identified through hand surgery-relevant keywords. Documents were processed using a hierarchical pipeline based on the RAPTOR methodology, which breaks down large texts into smaller segments to enhance accurate retrieval. The system was evaluated using 15 standardized clinical queries assessed using automated computational metrics for correctness and semantic similarity to source documents.

Results: The AI system demonstrated consistent performance with an average G-Eval correctness score of 0.79, SEM with an average similarity score of 0.75 (range: 0.54-0.86) and average maximum similarity score of 0.80 (range: 0.56-0.91), predominantly at moderate confidence levels. Generated recommendations were contextually appropriate and reliably linked to relevant hand surgery literature, providing accurate and clinically meaningful guidance.

Conclusion: The AI system, HandRAG, incorporating RAG and LLM approach offers potential benefits for evidence-based clinical decision support and education in hand surgery.