Evaluating the Capability of Large Language Model Chatbots for Generating Plain Language Summaries in Radiology

Abstract

Background

Plain language summary (PLS) are essential for making scientific research accessible to a broader audience. With the increasing capabilities of large language models (LLMs), there is the potential to automate the generation of PLS from complex scientific abstracts. This study assessed the performance of six LLM chatbots: ChatGPT, Claude, Copilot, Gemini, Meta AI, and Perplexity, in generating PLS from radiology research abstracts.

Methods

A total of 100 radiology abstracts were collected from PubMed. Six LLM chatbots were tasked with generating PLS for each abstract. Two expert radiologists independently evaluated the generated summaries for accuracy and readability, with their average scores being used for comparisons. Additionally, the Flesch–Kincaid (FK) grade level and Flesch reading ease score were applied to objectively assess readability.

Results

Comparisons of LLM-generated PLS revealed variations in both accuracy and readability across the models. Accuracy was highest for ChatGPT (4.94 ± 0.18) followed by Claude (4.75 ± 0.31). Readability was highest for ChatGPT (4.83 ± 0.27) followed by Perplexity (4.82 ± 0.29). The Flesch reading ease score was highest for Claude (62.53 ± 10.98) and lowest for ChatGPT (40.10 ± 11.24).

Conclusion

LLM chatbots show promise in the generation of PLS, but performance varies significantly between models in terms of both accuracy and readability. This study highlights the potential of LLMs to aid in science communication but underscores the need for careful model selection and human oversight.