Performance Change with the Ratio of Training Data A Case Study on the Binary Classification of COVID-19 Chest X-Ray by using Convolutional Neural Networks

Kuniki Imagawa, Kohei Shiomoto
{"title":"Performance Change with the Ratio of Training Data A Case Study on the Binary Classification of COVID-19 Chest X-Ray by using Convolutional Neural Networks","authors":"Kuniki Imagawa, Kohei Shiomoto","doi":"10.1109/MeMeA57477.2023.10171929","DOIUrl":null,"url":null,"abstract":"One of the features of artificial intelligence/machine learning-based medical devices resides in their ability to learn from real-world data. The performance may change after the market introduction. There are many aspects that contribute to the performance change relative to the real-world training data, such as the number and disease ratio. In actual clinical practice, the ratio of obtained training data varies from country to country, from region to region within each country, and from one hospital to another. Therefore, establishing a pre-change control plan at a premarket stage is essential to achieve safety and effectiveness through total product life cycles. In our previous work, we evaluated the performance change on the binary classification of coronavirus disease 2019 (COVID-19) and normal with the number of training data using two publicly large available chest X-ray (CXR) images. However, these results were obtained with the same ratio in the training data. Thus, this study aims to evaluate the performance change with a non-uniform ratio of COVID-19 CXR images based on the results of previous studies. We used the AlexNet and ResNet34 with and without the fine-tuning method as convolutional neural network (CNN) models. A total of 500, 1000, and 2000 CXR images were selected as training and validation datasets. These datasets represent states in which the performance change improves rapidly and those in which an equilibrium state is reached. Each dataset was divided into seven datasets, and the area under the curve was employed to evaluate the performance change for each dataset through independent 1000 test datasets with the same ratio. Our result shows that all performances indicate that there is an upward convex relationship to the ratio of COVID-19 CXR images, and the vertex is where the ratio is the same. This trend was remarkable for the rapidly improving state and the CNNs without a fine-tuning method. Moreover, the visual explanations technique called Grad-CAM for interpreting classification results of CNN models support these results.","PeriodicalId":191927,"journal":{"name":"2023 IEEE International Symposium on Medical Measurements and Applications (MeMeA)","volume":"110 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2023-06-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"2023 IEEE International Symposium on Medical Measurements and Applications (MeMeA)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/MeMeA57477.2023.10171929","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

Abstract

One of the features of artificial intelligence/machine learning-based medical devices resides in their ability to learn from real-world data. The performance may change after the market introduction. There are many aspects that contribute to the performance change relative to the real-world training data, such as the number and disease ratio. In actual clinical practice, the ratio of obtained training data varies from country to country, from region to region within each country, and from one hospital to another. Therefore, establishing a pre-change control plan at a premarket stage is essential to achieve safety and effectiveness through total product life cycles. In our previous work, we evaluated the performance change on the binary classification of coronavirus disease 2019 (COVID-19) and normal with the number of training data using two publicly large available chest X-ray (CXR) images. However, these results were obtained with the same ratio in the training data. Thus, this study aims to evaluate the performance change with a non-uniform ratio of COVID-19 CXR images based on the results of previous studies. We used the AlexNet and ResNet34 with and without the fine-tuning method as convolutional neural network (CNN) models. A total of 500, 1000, and 2000 CXR images were selected as training and validation datasets. These datasets represent states in which the performance change improves rapidly and those in which an equilibrium state is reached. Each dataset was divided into seven datasets, and the area under the curve was employed to evaluate the performance change for each dataset through independent 1000 test datasets with the same ratio. Our result shows that all performances indicate that there is an upward convex relationship to the ratio of COVID-19 CXR images, and the vertex is where the ratio is the same. This trend was remarkable for the rapidly improving state and the CNNs without a fine-tuning method. Moreover, the visual explanations technique called Grad-CAM for interpreting classification results of CNN models support these results.
基于卷积神经网络的COVID-19胸部x线图像二值分类研究
基于人工智能/机器学习的医疗设备的特点之一在于它们能够从现实世界的数据中学习。市场引入后,业绩可能会有所变化。与真实世界的训练数据相比,有许多方面会导致性能变化,例如数量和疾病比率。在实际的临床实践中,获得的培训数据的比例因国家而异,因地区而异,因地区而异,因医院而异。因此,在上市前阶段建立变更前控制计划对于在整个产品生命周期中实现安全性和有效性至关重要。在我们之前的工作中,我们使用两张公开的大型胸部x射线(CXR)图像,利用训练数据的数量评估了冠状病毒病2019 (COVID-19)和正常人二分类的性能变化。然而,这些结果是在训练数据中以相同的比率得到的。因此,本研究旨在基于以往研究结果,评估COVID-19 CXR图像比例不均匀时的性能变化。我们使用AlexNet和ResNet34作为卷积神经网络(CNN)模型。分别选取500、1000和2000张CXR图像作为训练和验证数据集。这些数据集代表性能变化迅速改善的状态和达到平衡状态的状态。将每个数据集划分为7个数据集,通过1000个独立的相同比例的测试数据集,采用曲线下面积来评估每个数据集的性能变化。我们的结果表明,所有性能都表明COVID-19 CXR图像的比例存在向上凸关系,并且顶点是比例相同的地方。这种趋势对于快速改善的状态和没有微调方法的cnn来说是值得注意的。此外,用于解释CNN模型分类结果的视觉解释技术Grad-CAM也支持了这些结果。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信