Artificial intelligence for solving pediatric clinical cases: A Retrieval-Augmented approach utilizing Llama3.2 and structured references

IF 4.1 2区医学 Q2 COMPUTER SCIENCE, INFORMATION SYSTEMS

International Journal of Medical Informatics Pub Date : 2025-06-23 DOI:10.1016/j.ijmedinf.2025.106027

Gianluca Mondillo, Simone Colosimo, Alessandra Perrotta, Vittoria Frattolillo, Mariapia Masino, Marco Martino, Emanuele Miraglia del Giudice, Pierluigi Marzuillo

{"title":"Artificial intelligence for solving pediatric clinical cases: A Retrieval-Augmented approach utilizing Llama3.2 and structured references","authors":"Gianluca Mondillo, Simone Colosimo, Alessandra Perrotta, Vittoria Frattolillo, Mariapia Masino, Marco Martino, Emanuele Miraglia del Giudice, Pierluigi Marzuillo","doi":"10.1016/j.ijmedinf.2025.106027","DOIUrl":null,"url":null,"abstract":"<div><h3>Background</h3><div>The “hallucinations” of Large Language Models (LLMs) raise concerns about their accuracy in pediatrics. This study aimed to evaluate whether integrating information from the Nelson Textbook of Pediatrics through a Retrieval-Augmented Generation (RAG) system could enhance the performance of Llama3.2 in addressing complex pediatric clinical cases.</div></div><div><h3>Methods</h3><div>We assessed the RAG system performance using 1,713 multiple-choice pediatric clinical questions from the MedQA dataset (n = 1,572) and Archives of Disease in Childhood–Education and Practice (n = 141). Each question was presented to Llama3.2 both in standalone mode and with RAG integration. The percentage of correct answers between models was compared using the chi-square test. p < 0.05 was considered statistically significant.</div></div><div><h3>Results</h3><div>The RAG-integrated system significantly outperformed standalone Llama3.2, achieving an overall accuracy of 67.78 % (1,161/1,713) compared to 46.18 % (791/1,713) for Llama3.2 alone (p = 1.5e-112). The improvement was consistent across all pediatric subspecialties.</div></div><div><h3>Conclusions</h3><div>Incorporating RAG systems into clinical decision-making can enhance reliability and safety.</div></div>","PeriodicalId":54950,"journal":{"name":"International Journal of Medical Informatics","volume":"203 ","pages":"Article 106027"},"PeriodicalIF":4.1000,"publicationDate":"2025-06-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"International Journal of Medical Informatics","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1386505625002448","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"COMPUTER SCIENCE, INFORMATION SYSTEMS","Score":null,"Total":0}

引用次数: 0

Abstract

Background

The “hallucinations” of Large Language Models (LLMs) raise concerns about their accuracy in pediatrics. This study aimed to evaluate whether integrating information from the Nelson Textbook of Pediatrics through a Retrieval-Augmented Generation (RAG) system could enhance the performance of Llama3.2 in addressing complex pediatric clinical cases.

Methods

We assessed the RAG system performance using 1,713 multiple-choice pediatric clinical questions from the MedQA dataset (n = 1,572) and Archives of Disease in Childhood–Education and Practice (n = 141). Each question was presented to Llama3.2 both in standalone mode and with RAG integration. The percentage of correct answers between models was compared using the chi-square test. p < 0.05 was considered statistically significant.

Results

The RAG-integrated system significantly outperformed standalone Llama3.2, achieving an overall accuracy of 67.78 % (1,161/1,713) compared to 46.18 % (791/1,713) for Llama3.2 alone (p = 1.5e-112). The improvement was consistent across all pediatric subspecialties.

Conclusions

Incorporating RAG systems into clinical decision-making can enhance reliability and safety.

查看原文本刊更多论文

人工智能解决儿科临床病例：利用Llama3.2和结构化参考文献的检索增强方法

大语言模型（llm）的“幻觉”引起了人们对其在儿科中的准确性的关注。本研究旨在评估通过检索-增强生成（RAG）系统整合尼尔森儿科教科书信息是否可以提高Llama3.2处理复杂儿科临床病例的性能。方法我们使用来自MedQA数据集（n = 1572）和儿童教育与实践疾病档案（n = 141）的1713个儿科临床选择题来评估RAG系统的性能。每个问题都以独立模式和与RAG集成的方式呈现给Llama3.2。模型间正确答案的百分比采用卡方检验进行比较。p & lt;0.05认为有统计学意义。结果rag -集成系统的总体准确率为67.78%(1161 / 1713)，明显优于单独使用Llama3.2的46.18% (791/ 1713)（p = 1.5e-112）。所有儿科亚专科的改善是一致的。结论将RAG系统纳入临床决策可提高决策的可靠性和安全性。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

International Journal of Medical Informatics 医学-计算机：信息系统

CiteScore

8.90

自引率

4.10%

发文量

217

审稿时长

42 days

期刊介绍： International Journal of Medical Informatics provides an international medium for dissemination of original results and interpretative reviews concerning the field of medical informatics. The Journal emphasizes the evaluation of systems in healthcare settings. The scope of journal covers: Information systems, including national or international registration systems, hospital information systems, departmental and/or physician''s office systems, document handling systems, electronic medical record systems, standardization, systems integration etc.; Computer-aided medical decision support systems using heuristic, algorithmic and/or statistical methods as exemplified in decision theory, protocol development, artificial intelligence, etc. Educational computer based programs pertaining to medical informatics or medicine in general; Organizational, economic, social, clinical impact, ethical and cost-benefit aspects of IT applications in health care.