AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science最新文献_第10页

Avoiding Biased Clinical Machine Learning Model Performance Estimates in the Presence of Label Selection. 避免标签选择情况下有偏差的临床机器学习模型性能评估

AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science Pub Date : 2023-06-16 eCollection Date: 2023-01-01

Conor K Corbin, Michael Baiocchi, Jonathan H Chen

{"title":"Avoiding Biased Clinical Machine Learning Model Performance Estimates in the Presence of Label Selection.","authors":"Conor K Corbin, Michael Baiocchi, Jonathan H Chen","doi":"","DOIUrl":"","url":null,"abstract":"When evaluating the performance of clinical machine learning models, one must consider the deployment population. When the population of patients with observed labels is only a subset of the deployment population (label selection), standard model performance estimates on the observed population may be misleading. In this study we describe three classes of label selection and simulate five causally distinct scenarios to assess how particular selection mechanisms bias a suite of commonly reported binary machine learning model performance metrics. Simulations reveal that when selection is affected by observed features, naive estimates of model discrimination may be misleading. When selection is affected by labels, naive estimates of calibration fail to reflect reality. We borrow traditional weighting estimators from causal inference literature and find that when selection probabilities are properly specified, they recover full population estimates. We then tackle the real-world task of monitoring the performance of deployed machine learning models whose interactions with clinicians feed-back and affect the selection mechanism of the labels. We train three machine learning models to flag low-yield laboratory diagnostics, and simulate their intended consequence of reducing wasteful laboratory utilization. We find that naive estimates of AUROC on the observed population undershoot actual performance by up to 20%. Such a disparity could be large enough to lead to the wrongful termination of a successful clinical decision support tool. We propose an altered deployment procedure, one that combines injected randomization with traditional weighted estimates, and find it recovers true model performance.","PeriodicalId":72181,"journal":{"name":"AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-06-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10283136/pdf/2405.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"9703649","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Linking Ambient NO2 Pollution Measures with Electronic Health Record Data to Study Asthma Exacerbations. 将环境二氧化氮污染测量与电子健康记录数据联系起来研究哮喘恶化。

AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science Pub Date : 2023-06-16 eCollection Date: 2023-01-01

Alana Schreibman, Sherrie Xie, Rebecca A Hubbard, Blanca E Himes

{"title":"Linking Ambient NO2 Pollution Measures with Electronic Health Record Data to Study Asthma Exacerbations.","authors":"Alana Schreibman, Sherrie Xie, Rebecca A Hubbard, Blanca E Himes","doi":"","DOIUrl":"","url":null,"abstract":"Electronic health record (EHR)-derived data can be linked to geospatially distributed socioeconomic and environmental factors to conduct large-scale epidemiologic studies. Ambient NO2 is a known environmental risk factor for asthma. However, health exposure studies often rely on data from geographically sparse regulatory monitors that may not reflect true individual exposure. We contrasted use of interpolated NO2 regulatory monitor data with raw satellite measurements and satellite-derived ground estimates, building on previous work which has computed improved exposure estimates from remotely sensed data. Raw satellite and satellite-derived ground measurements captured spatial variation missed by interpolated ground monitor measurements. Multivariable analyses comparing these three NO2 measurement approaches (interpolated monitor, raw satellite, and satellite-derived) revealed a positive relationship between exposure and asthma exacerbations for both satellite measurements. Exposure-outcome relationships using the interpolated monitor NO2 were inconsistent with known relationships to asthma, suggesting that interpolated monitor data might yield misleading results in small region studies.","PeriodicalId":72181,"journal":{"name":"AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-06-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10283087/pdf/2145.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"9832116","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

TRESTLE: Toolkit for Reproducible Execution of Speech, Text and Language Experiments. TRESTLE：可重复执行语音、文本和语言实验的工具包。

AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science Pub Date : 2023-06-16 eCollection Date: 2023-01-01

Changye Li, Weizhe Xu, Trevor Cohen, Martin Michalowski, Serguei Pakhomov

{"title":"TRESTLE: Toolkit for Reproducible Execution of Speech, Text and Language Experiments.","authors":"Changye Li, Weizhe Xu, Trevor Cohen, Martin Michalowski, Serguei Pakhomov","doi":"","DOIUrl":"","url":null,"abstract":"The evidence is growing that machine and deep learning methods can learn the subtle differences between the language produced by people with various forms of cognitive impairment such as dementia and cognitively healthy individuals. Valuable public data repositories such as TalkBank have made it possible for researchers in the computational community to join forces and learn from each other to make significant advances in this area. However, due to variability in approaches and data selection strategies used by various researchers, results obtained by different groups have been difficult to compare directly. In this paper, we present TRESTLE (Toolkit for Reproducible Execution of Speech Text and Language Experiments), an open source platform that focuses on two datasets from the TalkBank repository with dementia detection as an illustrative domain. Successfully deployed in the hackallenge (Hackathon/Challenge) of the International Workshop on Health Intelligence at AAAI 2022, TRESTLE provides a precise digital blueprint of the data pre-processing and selection strategies that can be reused via TRESTLE by other researchers seeking comparable results with their peers and current state-of-the-art (SOTA) approaches.","PeriodicalId":72181,"journal":{"name":"AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-06-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10283131/pdf/2277.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"9715633","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Principal Investigators' Perceptions on Factors Associated with Successful Recruitment in Clinical Trials. 主要研究人员对临床试验成功招募相关因素的看法。

AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science Pub Date : 2023-06-16 eCollection Date: 2023-01-01

Betina Idnay, Alex Butler, Yilu Fang, Ziran Li, Junghwan Lee, Casey Ta, Cong Liu, Brenda Ruotolo, Chi Yuan, Huanyao Chen, George Hripcsak, Elaine Larson, Chunhua Weng

{"title":"Principal Investigators' Perceptions on Factors Associated with Successful Recruitment in Clinical Trials.","authors":"Betina Idnay, Alex Butler, Yilu Fang, Ziran Li, Junghwan Lee, Casey Ta, Cong Liu, Brenda Ruotolo, Chi Yuan, Huanyao Chen, George Hripcsak, Elaine Larson, Chunhua Weng","doi":"","DOIUrl":"","url":null,"abstract":"Participant recruitment continues to be a challenge to the success of randomized controlled trials, resulting in increased costs, extended trial timelines and delayed treatment availability. Literature provides evidence that study design features (e.g., trial phase, study site involvement) and trial sponsor are significantly associated with recruitment success. Principal investigators oversee the conduct of clinical trials, including recruitment. Through a cross-sectional survey and a thematic analysis of free-text responses, we assessed the perceptions of sixteen principal investigators regarding success factors for participant recruitment. Study site involvement and funding source do not necessarily make recruitment easier or more challenging from the perspective of the principal investigators. The most commonly used recruitment strategies are also the most effort inefficient (e.g., in-person recruitment, reviewing the electronic medical records for prescreening). Finally, we recommended actionable steps, such as improving staff support and leveraging informatics-driven approaches, to allow clinical researchers to enhance participant recruitment.","PeriodicalId":72181,"journal":{"name":"AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-06-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10283115/pdf/2207.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"10070909","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Efficient Federated Kinship Relationship Identification. 有效的联邦亲属关系识别。

AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science Pub Date : 2023-06-16 eCollection Date: 2023-01-01

Xinyue Wang, Leonard Dervishi, Wentao Li, Xiaoqian Jiang, Erman Ayday, Jaideep Vaidya

引用次数: 0

Using Explainable AI to Cross-Validate Socio-economic Disparities Among Covid-19 Patient Mortality 使用可解释的人工智能交叉验证Covid-19患者死亡率的社会经济差异

AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science Pub Date : 2023-02-16 DOI: 10.48550/arXiv.2302.08605

Linlin Shi, Redoan Rahman, E. Melamed, J. Gwizdka, Justin F. Rousseau, Ying Ding

{"title":"Using Explainable AI to Cross-Validate Socio-economic Disparities Among Covid-19 Patient Mortality","authors":"Linlin Shi, Redoan Rahman, E. Melamed, J. Gwizdka, Justin F. Rousseau, Ying Ding","doi":"10.48550/arXiv.2302.08605","DOIUrl":"https://doi.org/10.48550/arXiv.2302.08605","url":null,"abstract":"This paper applies eXplainable Artificial Intelligence (XAI) methods to investigate the socioeconomic disparities in COVID-19 patient mortality. An Extreme Gradient Boosting (XGBoost) prediction model is built based on a de-identified Austin area hospital dataset to predict the mortality of COVID-19 patients. We apply two XAI methods, Shapley Additive exPlanations (SHAP) and Locally Interpretable Model Agnostic Explanations (LIME), to compare the global and local interpretation of feature importance. This paper demonstrates the advantages of using XAI which shows the feature importance and decisive capability. Furthermore, we use the XAI methods to cross-validate their interpretations for individual patients. The XAI models reveal that Medicare financial class, older age, and gender have high impact on the mortality prediction. We find that LIME's local interpretation does not show significant differences in feature importance comparing to SHAP, which suggests pattern confirmation. This paper demonstrates the importance of XAI methods in cross-validation of feature attributions.","PeriodicalId":72181,"journal":{"name":"AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-02-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"83219973","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

TRESTLE: Toolkit for Reproducible Execution of Speech, Text and Language Experiments TRESTLE:语音，文本和语言实验的可重复执行工具包

AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science Pub Date : 2023-02-14 DOI: 10.48550/arXiv.2302.07322

Changye Li, T. Cohen, Martin Michalowski, Serguei V. S. Pakhomov

{"title":"TRESTLE: Toolkit for Reproducible Execution of Speech, Text and Language Experiments","authors":"Changye Li, T. Cohen, Martin Michalowski, Serguei V. S. Pakhomov","doi":"10.48550/arXiv.2302.07322","DOIUrl":"https://doi.org/10.48550/arXiv.2302.07322","url":null,"abstract":"The evidence is growing that machine and deep learning methods can learn the subtle differences between the language produced by people with various forms of cognitive impairment such as dementia and cognitively healthy individuals. Valuable public data repositories such as TalkBank have made it possible for researchers in the computational community to join forces and learn from each other to make significant advances in this area. However, due to variability in approaches and data selection strategies used by various researchers, results obtained by different groups have been difficult to compare directly. In this paper, we present TRESTLE (Toolkit for Reproducible Execution of Speech Text and Language Experiments), an open source platform that focuses on two datasets from the TalkBank repository with dementia detection as an illustrative domain. Successfully deployed in the hackallenge (Hackathon/Challenge) of the International Workshop on Health Intelligence at AAAI 2022, TRESTLE provides a precise digital blueprint of the data pre-processing and selection strategies that can be reused via TRESTLE by other researchers seeking comparable results with their peers and current state-of-the-art (SOTA) approaches.","PeriodicalId":72181,"journal":{"name":"AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-02-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"87266086","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Evaluate underdiagnosis and overdiagnosis bias of deep learning model on primary open-angle glaucoma diagnosis in under-served patient populations 评估深度学习模型在缺医少药人群中原发性开角型青光眼诊断中的漏诊和过度诊断偏差

AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science Pub Date : 2023-01-26 DOI: 10.48550/arXiv.2301.11315

Mingquan Lin, Yuyun Xiao, Bojian Hou, Tingyi Wanyan, M. Sharma, Zhangyang Wang, Fei Wang, S. V. Tassel, Yifan Peng

{"title":"Evaluate underdiagnosis and overdiagnosis bias of deep learning model on primary open-angle glaucoma diagnosis in under-served patient populations","authors":"Mingquan Lin, Yuyun Xiao, Bojian Hou, Tingyi Wanyan, M. Sharma, Zhangyang Wang, Fei Wang, S. V. Tassel, Yifan Peng","doi":"10.48550/arXiv.2301.11315","DOIUrl":"https://doi.org/10.48550/arXiv.2301.11315","url":null,"abstract":"In the United States, primary open-angle glaucoma (POAG) is the leading cause of blindness, especially among African American and Hispanic individuals. Deep learning has been widely used to detect POAG using fundus images as its performance is comparable to or even surpasses diagnosis by clinicians. However, human bias in clinical diagnosis may be reflected and amplified in the widely-used deep learning models, thus impacting their performance. Biases may cause (1) underdiagnosis, increasing the risks of delayed or inadequate treatment, and (2) overdiagnosis, which may increase individuals' stress, fear, well-being, and unnecessary/costly treatment. In this study, we examined the underdiagnosis and overdiagnosis when applying deep learning in POAG detection based on the Ocular Hypertension Treatment Study (OHTS) from 22 centers across 16 states in the United States. Our results show that the widely-used deep learning model can underdiagnose or overdiagnose under-served populations. The most underdiagnosed group is female younger (< 60 yrs) group, and the most overdiagnosed group is Black older (≥ 60 yrs) group. Biased diagnosis through traditional deep learning methods may delay disease detection, treatment and create burdens among under-served populations, thereby, raising ethical concerns about using deep learning models in ophthalmology clinics.","PeriodicalId":72181,"journal":{"name":"AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-01-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"76296744","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1

Discovering Precision AD Biomarkers with Varying Prognosis Effects in Genetics Driven Subpopulations. 在遗传驱动的亚群中发现具有不同预后影响的精准AD生物标志物。

AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science Pub Date : 2023-01-01

Brian N Lee, Junwen Wang, Kwangsik Nho, Andrew J Saykin, Li Shen

{"title":"Discovering Precision AD Biomarkers with Varying Prognosis Effects in Genetics Driven Subpopulations.","authors":"Brian N Lee, Junwen Wang, Kwangsik Nho, Andrew J Saykin, Li Shen","doi":"","DOIUrl":"","url":null,"abstract":"Alzheimer's Disease (AD) is a highly heritable neurodegenerative disorder characterized by memory impairments. Understanding how genetic factors contribute to AD pathology may inform interventions to slow or prevent the progression of AD. We performed stratified genetic analyses of 1,574 Alzheimer's Disease Neuroimaging Initiative (ADNI) participants to examine associations between levels of quantitative traits (QT's) and future diagnosis. The Chow test was employed to determine if an individual's genetic profile affects identified predictive relationships between QT's and future diagnosis. Our chow test analysis discovered that cognitive and PET-based biomarkers differentially predicted future diagnosis when stratifying on allelic dosage of AD loci. Post-hoc bootstrapped and association analyses of biomarkers confirmed differential effects, emphasizing the necessity of stratified models to realize individualized AD diagnosis prediction. This novel application of the Chow test allows for the quantification and direct comparison of genetic-based differences. Our findings, as well as the identified QT-future diagnosis relationships, warrant future investigation from a biological context.","PeriodicalId":72181,"journal":{"name":"AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science","volume":null,"pages":null},"PeriodicalIF":0.0,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10283147/pdf/2152.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"10070915","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Assessing the Predictive and Analytics Capability of Electronic Clinical Data for High-Cost Patients. 评估高成本患者电子临床数据的预测和分析能力。

AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science Pub Date : 2023-01-01

Saathvika Diviti, Adam Wilcox

引用次数: 0