Configuring large language models to deliver patient-facing explanations of pathology reports.

Abstract

Objectives: Timely access to pathology reports has increased the need for clear patient-facing explanations. We evaluated whether large language model (LLM)-generated responses to pathology report questions from patients are comparable in quality to explanations written by pathologists and assessed how LLM configuration influences performance.

Methods: Sixty-five anonymized real-world patient questions from an online pathology education platform were answered using 5 LLM configurations varying by model architecture, prompting strategy, and retrieval-augmented generation. Responses were evaluated using a structured rubric that assessed accuracy, relevance, clarity, empathy, and safety; they were compared using pairwise arena testing with Bradley-Terry modeling to rank performance.

Results: Across rubric domains, LLM responses demonstrated performance comparable to pathologist explanations, with 1 configuration meeting noninferiority criteria. Pairwise arena comparisons indicated that the configuration parameters strongly influenced performance, with both model size and retrieval augmentation associated with improved response preference. The highest-performing configuration combined a larger model with retrieval from a curated pathology knowledge base and was strongly preferred over pathologist-written responses.

Conclusions: Carefully configured LLM systems can generate patient-facing explanations of pathology reports comparable in quality to pathologist-written explanations. Prompting strategy, model size, and retrieval integration were associated with performance differences, underscoring the importance of system configuration in developing LLM-based tools to expand access to understandable pathology information.