Multi-Institutional Evaluation and Training of Breast Density Classification AI Algorithm Using ACR Connect and AI-LAB

IF 4 3区医学 Q1 RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING

Journal of the American College of Radiology Pub Date : 2025-02-01 DOI:10.1016/j.jacr.2024.11.003

Laura Brink , Ricardo Amaya Romero , Laura Coombs , Mike Tilkin , Sina Mazaheri MD , Judy Gichoya MD , Zachary Zaiman MD , Hari Trivedi MD , Adam Medina MD , Bernardo C. Bizzo MD, PhD , Ken Chang MD , Jayashree Kalpathy-Cramer MD , Mannudeep K. Kalra MD , Bruno Astuto MD , Carolina Ramirez MD , Sharmila Majumdar MD , Amie Y. Lee MD , Christoph I. Lee MD, MS, MBA , Nathan M. Cross MD, MS , Po-Hao Chen MD , Christoph Wald MD

{"title":"Multi-Institutional Evaluation and Training of Breast Density Classification AI Algorithm Using ACR Connect and AI-LAB","authors":"Laura Brink , Ricardo Amaya Romero , Laura Coombs , Mike Tilkin , Sina Mazaheri MD , Judy Gichoya MD , Zachary Zaiman MD , Hari Trivedi MD , Adam Medina MD , Bernardo C. Bizzo MD, PhD , Ken Chang MD , Jayashree Kalpathy-Cramer MD , Mannudeep K. Kalra MD , Bruno Astuto MD , Carolina Ramirez MD , Sharmila Majumdar MD , Amie Y. Lee MD , Christoph I. Lee MD, MS, MBA , Nathan M. Cross MD, MS , Po-Hao Chen MD , Christoph Wald MD","doi":"10.1016/j.jacr.2024.11.003","DOIUrl":null,"url":null,"abstract":"<div><h3>Objective</h3><div>To demonstrate and test the capabilities of the ACR Connect and AI-LAB software platform by implementing multi-institutional artificial intelligence (AI) training and validation for breast density classification.</div></div><div><h3>Methods</h3><div>In this proof-of-concept study, six US-based hospitals installed Connect and AI-LAB. A breast density algorithm was trained and tested on retrospective mammograms. We recorded time to receive institutional review board approval, to install software locally, and to complete the testing and training. We calculated the performance of the breast density algorithm at each participating hospital and compared it to the performance of a holdout multi-institutional clinical trial testing dataset and a retrospective multi-institutional dataset. We calculated the performance of the locally fine-tuned models on the holdout test datasets.</div></div><div><h3>Results</h3><div>The median time to receive institutional review board approval was 66 days, and the median time to successfully install Connect and AI-LAB locally was 157 days. The median time to complete breast density algorithm testing and training was 216 days. The breast density algorithm performed worse at each hospital than on the holdout test dataset, suggesting poor generalizability of the base model. The fine-tuned models had mixed performance locally and performed poorly on the test dataset.</div></div><div><h3>Discussion</h3><div>In this study, we demonstrate the successful installation and implementation of Connect and AI-LAB software platforms at six facilities using a breast density algorithm. Our results suggest poor generalizability of an algorithm trained on a single dataset and algorithms fine-tuned at individual institutions, emphasizing the hypothetical importance of multi-institutional testing and training.</div></div>","PeriodicalId":49044,"journal":{"name":"Journal of the American College of Radiology","volume":"22 2","pages":"Pages 211-219"},"PeriodicalIF":4.0000,"publicationDate":"2025-02-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of the American College of Radiology","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1546144024009128","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING","Score":null,"Total":0}

引用次数: 0

Abstract

Objective

To demonstrate and test the capabilities of the ACR Connect and AI-LAB software platform by implementing multi-institutional artificial intelligence (AI) training and validation for breast density classification.

Methods

In this proof-of-concept study, six US-based hospitals installed Connect and AI-LAB. A breast density algorithm was trained and tested on retrospective mammograms. We recorded time to receive institutional review board approval, to install software locally, and to complete the testing and training. We calculated the performance of the breast density algorithm at each participating hospital and compared it to the performance of a holdout multi-institutional clinical trial testing dataset and a retrospective multi-institutional dataset. We calculated the performance of the locally fine-tuned models on the holdout test datasets.

Results

The median time to receive institutional review board approval was 66 days, and the median time to successfully install Connect and AI-LAB locally was 157 days. The median time to complete breast density algorithm testing and training was 216 days. The breast density algorithm performed worse at each hospital than on the holdout test dataset, suggesting poor generalizability of the base model. The fine-tuned models had mixed performance locally and performed poorly on the test dataset.

Discussion

In this study, we demonstrate the successful installation and implementation of Connect and AI-LAB software platforms at six facilities using a breast density algorithm. Our results suggest poor generalizability of an algorithm trained on a single dataset and algorithms fine-tuned at individual institutions, emphasizing the hypothetical importance of multi-institutional testing and training.

查看原文本刊更多论文

使用 ACR Connect 和 AI-LAB 对乳腺密度分类人工智能算法进行多机构评估和训练。

目的通过对乳腺密度分类进行多机构人工智能（AI）培训和验证，展示并测试美国放射学会（ACR）Connect 和 AI-LAB 软件平台的能力：在这项概念验证研究中，六家美国医院安装了 Connect 和 AI-LAB。在回顾性乳房 X 光照片上对乳腺密度算法进行了训练和测试。我们记录了获得 IRB 批准、在本地安装软件以及完成测试和培训所需的时间。我们计算了每家参与医院的乳腺密度算法的性能，并将其与暂缓执行的多机构临床试验测试数据集和回顾性多机构数据集的性能进行了比较。我们计算了局部微调模型在暂缓测试数据集上的性能：结果：获得 IRB 批准的中位时间为 66 天，在本地成功安装 Connect 和 AI-LAB 的中位时间为 157 天。完成乳腺密度算法测试和训练的中位时间为 216 天。乳腺密度算法在每家医院的表现都比在暂缓测试数据集上的表现差，这表明基础模型的通用性较差。微调模型在本地的表现参差不齐，在测试数据集上的表现较差：在本研究中，我们利用乳腺密度算法展示了 Connect 和 AI-LAB 软件平台在六家机构的成功安装和实施。我们的结果表明，在单个数据集上训练的算法和在单个机构微调的算法的通用性很差，这强调了多机构测试和训练的假设重要性。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of the American College of Radiology RADIOLOGY, NUCLEAR MEDICINE & MEDICAL IMAGING-

CiteScore

6.30

自引率

8.90%

发文量

312

审稿时长

34 days

期刊介绍： The official journal of the American College of Radiology, JACR informs its readers of timely, pertinent, and important topics affecting the practice of diagnostic radiologists, interventional radiologists, medical physicists, and radiation oncologists. In so doing, JACR improves their practices and helps optimize their role in the health care system. By providing a forum for informative, well-written articles on health policy, clinical practice, practice management, data science, and education, JACR engages readers in a dialogue that ultimately benefits patient care.