ChatGPT: Evaluating answers on contrast media related questions and finetuning by providing the model with the ESUR guideline on contrast agents

Abstract

Objective

This study aimed to assess the feasibility of GPT-4 for answering questions related to contrast media with and without the context of the European Society of Urogenital Radiology (ESUR) guideline on contrast agents. The overarching goal was to determine whether contextual enrichment by providing guideline information improves answers of GPT-4 for clinical decision-making in radiology.

Methods

A set of 64 questions, based on the ESUR guideline on contrast agents mirroring pertinent sections, was developed and posed to GPT-4 both directly and after providing the guideline using a plugin. Responses were graded by experienced radiologists for quality of information and accuracy in pinpointing information from the guideline as well as by radiology residents for utility, using Likert-scales.

Results

GPT-4′s performance improved significantly with the guideline. Without the guideline, average quality rating was 3.98, which increased to 4.33 with the guideline (p = 0036). In terms of accuracy, 82.3% of answers matched the information from the guideline. Utility scores also reflected a significant improvement with the guideline, with average scores of 4.1 (without) and 4.4 (with) (p = 0.008) with a Fleiss´ Kappa of 0.44.

Conclusion

GPT-4, when contextually enriched with a guideline, demonstrates enhanced capability in providing guideline-backed recommendations. This approach holds promise for real-time clinical decision-support, making guidelines more actionable. However, further refinements are necessary to maximize the potential of large language models (LLMs). Inherent limitations need to be addressed.