RTPhy-ChatBot: A RAG-Based intelligent assistant for radiotherapy physics using LLaMA3 and AAPM reports

IF 2.2 4区医学 Q3 RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING

Journal of Applied Clinical Medical Physics Pub Date : 2025-10-09 DOI:10.1002/acm2.70263

Shuoyang Wei, Ankang Hu, Zhiqun Wang, Xiangyin Meng, Lang Yu, Bo Yang, Jie Qiu

{"title":"RTPhy-ChatBot: A RAG-Based intelligent assistant for radiotherapy physics using LLaMA3 and AAPM reports","authors":"Shuoyang Wei, Ankang Hu, Zhiqun Wang, Xiangyin Meng, Lang Yu, Bo Yang, Jie Qiu","doi":"10.1002/acm2.70263","DOIUrl":null,"url":null,"abstract":"<div>\n \n \n <section>\n \n <h3> Background</h3>\n \n <p>Medical physics plays a crucial role in radiotherapy, with ongoing technological advancements aimed at improving treatment outcomes. However, the rapid pace of innovation presents challenges for medical physicists, who must continuously acquire and integrate complex information for effective decision-making and communication.</p>\n </section>\n \n <section>\n \n <h3> Purpose</h3>\n \n <p>To support efficient knowledge acquisition, we developed RTPhy-ChatBot, an intelligent assistant tailored to radiotherapy physics. The objective was to create a reliable and precise tool to assist medical physicists in their daily work.</p>\n </section>\n \n <section>\n \n <h3> Methods</h3>\n \n <p>The knowledge base for RTPhy-ChatBot was constructed from publications by the American Association of Physicists in Medicine (AAPM), which were converted into markdown format, segmented, and embedded using the bge-base-en-v1.5 model. RTPhy-ChatBot employed the Meta-LLaMA3-8B-Instruct model for response generation. We compared its performance with several commercial large language models (LLMs) across 20 template questions and evaluated the impact of zero-shot chain-of-thought (CoT) reasoning. In addition to expert scoring by senior medical physicists, we conducted Rouge score analysis against synthesized reference answers.</p>\n </section>\n \n <section>\n \n <h3> Results</h3>\n \n <p>RTPhy-ChatBot demonstrated strong performance in answering radiotherapy physics questions. Across 20 questions, it achieved an average score of 4.0 ± 0.9, compared to 3.9 ± 1.1 for Gemini-2.0-Flash, 4.0 ± 1.4 for GPT-4o, and 3.8 ± 1.2 for Moonshot-v1. It excelled in questions involving specific quality assurance standards. Rouge analysis yielded scores of 0.5127 (Rouge-1), 0.2119 (Rouge-2), and 0.2748 (Rouge-L), closely matching commercial LLMs.</p>\n </section>\n \n <section>\n \n <h3> Conclusions</h3>\n \n <p>RTPhy-ChatBot proved to be an effective intelligent assistant for radiotherapy physics, delivering accurate, referenced responses grounded in AAPM publications. Despite lacking online access, it matched or exceeded the performance of commercial LLMs in domain-specific tasks. This pilot study highlights the potential of domain-specific assistants in supporting clinical workflows.</p>\n </section>\n </div>","PeriodicalId":14989,"journal":{"name":"Journal of Applied Clinical Medical Physics","volume":"26 10","pages":""},"PeriodicalIF":2.2000,"publicationDate":"2025-10-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://aapm.onlinelibrary.wiley.com/doi/epdf/10.1002/acm2.70263","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Applied Clinical Medical Physics","FirstCategoryId":"3","ListUrlMain":"https://aapm.onlinelibrary.wiley.com/doi/10.1002/acm2.70263","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING","Score":null,"Total":0}

引用次数: 0

Abstract

Background

Medical physics plays a crucial role in radiotherapy, with ongoing technological advancements aimed at improving treatment outcomes. However, the rapid pace of innovation presents challenges for medical physicists, who must continuously acquire and integrate complex information for effective decision-making and communication.

Purpose

To support efficient knowledge acquisition, we developed RTPhy-ChatBot, an intelligent assistant tailored to radiotherapy physics. The objective was to create a reliable and precise tool to assist medical physicists in their daily work.

Methods

The knowledge base for RTPhy-ChatBot was constructed from publications by the American Association of Physicists in Medicine (AAPM), which were converted into markdown format, segmented, and embedded using the bge-base-en-v1.5 model. RTPhy-ChatBot employed the Meta-LLaMA3-8B-Instruct model for response generation. We compared its performance with several commercial large language models (LLMs) across 20 template questions and evaluated the impact of zero-shot chain-of-thought (CoT) reasoning. In addition to expert scoring by senior medical physicists, we conducted Rouge score analysis against synthesized reference answers.

Results

RTPhy-ChatBot demonstrated strong performance in answering radiotherapy physics questions. Across 20 questions, it achieved an average score of 4.0 ± 0.9, compared to 3.9 ± 1.1 for Gemini-2.0-Flash, 4.0 ± 1.4 for GPT-4o, and 3.8 ± 1.2 for Moonshot-v1. It excelled in questions involving specific quality assurance standards. Rouge analysis yielded scores of 0.5127 (Rouge-1), 0.2119 (Rouge-2), and 0.2748 (Rouge-L), closely matching commercial LLMs.

Conclusions

RTPhy-ChatBot proved to be an effective intelligent assistant for radiotherapy physics, delivering accurate, referenced responses grounded in AAPM publications. Despite lacking online access, it matched or exceeded the performance of commercial LLMs in domain-specific tasks. This pilot study highlights the potential of domain-specific assistants in supporting clinical workflows.

Abstract Image

查看原文本刊更多论文

RTPhy-ChatBot：基于拉格的放疗物理智能助手，使用LLaMA3和AAPM报告。

背景：医学物理学在放射治疗中起着至关重要的作用，不断的技术进步旨在改善治疗结果。然而，创新的快速步伐给医学物理学家带来了挑战，他们必须不断获取和整合复杂的信息，以进行有效的决策和沟通。目的：为了支持有效的知识获取，我们开发了RTPhy-ChatBot，一种针对放射物理的智能助手。其目标是创建一个可靠和精确的工具，以协助医学物理学家的日常工作。方法：RTPhy-ChatBot知识库以美国医学物理学家协会（American Association of physics in Medicine， AAPM）的出版物为基础构建，采用big -base-en-v1.5模型将其转换为markdown格式，进行分割和嵌入。RTPhy-ChatBot采用meta - llama3 - 8b - directive模型生成响应。我们通过20个模板问题将其性能与几个商业大型语言模型（llm）进行了比较，并评估了零射击思维链（CoT）推理的影响。除了由资深医学物理学家进行专家评分外，我们还对综合参考答案进行了Rouge评分分析。结果：RTPhy-ChatBot在回答放疗物理问题方面表现出色。在20个问题中，它的平均得分为4.0±0.9，而Gemini-2.0-Flash的平均得分为3.9±1.1，gpt - 40的平均得分为4.0±1.4，Moonshot-v1的平均得分为3.8±1.2。它在涉及具体质量保证标准的问题上表现出色。Rouge分析的得分为0.5127 （Rouge-1）、0.2119 （Rouge-2）和0.2748 (Rouge- l)，与商业llm非常匹配。结论：RTPhy-ChatBot被证明是放疗物理的有效智能助手，提供基于AAPM出版物的准确，参考的响应。尽管缺乏在线访问，但它在特定领域任务中的表现与商业法学硕士相当甚至超过。这项试点研究强调了领域特定助手在支持临床工作流程方面的潜力。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of Applied Clinical Medical Physics 医学-核医学

CiteScore

3.60

自引率

19.00%

发文量

331

审稿时长

3 months

期刊介绍： Journal of Applied Clinical Medical Physics is an international Open Access publication dedicated to clinical medical physics. JACMP welcomes original contributions dealing with all aspects of medical physics from scientists working in the clinical medical physics around the world. JACMP accepts only online submission. JACMP will publish: -Original Contributions: Peer-reviewed, investigations that represent new and significant contributions to the field. Recommended word count: up to 7500. -Review Articles: Reviews of major areas or sub-areas in the field of clinical medical physics. These articles may be of any length and are peer reviewed. -Technical Notes: These should be no longer than 3000 words, including key references. -Letters to the Editor: Comments on papers published in JACMP or on any other matters of interest to clinical medical physics. These should not be more than 1250 (including the literature) and their publication is only based on the decision of the editor, who occasionally asks experts on the merit of the contents. -Book Reviews: The editorial office solicits Book Reviews. -Announcements of Forthcoming Meetings: The Editor may provide notice of forthcoming meetings, course offerings, and other events relevant to clinical medical physics. -Parallel Opposed Editorial: We welcome topics relevant to clinical practice and medical physics profession. The contents can be controversial debate or opposed aspects of an issue. One author argues for the position and the other against. Each side of the debate contains an opening statement up to 800 words, followed by a rebuttal up to 500 words. Readers interested in participating in this series should contact the moderator with a proposed title and a short description of the topic