JAMIA Open最新文献

{"title":"Algorithmic individual fairness and healthcare: a scoping review.","authors":"Joshua W Anderson, Shyam Visweswaran","doi":"10.1093/jamiaopen/ooae149","DOIUrl":"10.1093/jamiaopen/ooae149","url":null,"abstract":"<p><strong>Objectives: </strong>Statistical and artificial intelligence algorithms are increasingly being developed for use in healthcare. These algorithms may reflect biases that magnify disparities in clinical care, and there is a growing need for understanding how algorithmic biases can be mitigated in pursuit of algorithmic fairness. We conducted a scoping review on algorithmic individual fairness (IF) to understand the current state of research in the metrics and methods developed to achieve IF and their applications in healthcare.</p><p><strong>Materials and methods: </strong>We searched four databases: PubMed, ACM Digital Library, IEEE Xplore, and medRxiv for algorithmic IF metrics, algorithmic bias mitigation, and healthcare applications. Our search was restricted to articles published between January 2013 and November 2024. We identified 2498 articles through database searches and seven additional articles, of which 32 articles were included in the review. Data from the selected articles were extracted, and the findings were synthesized.</p><p><strong>Results: </strong>Based on the 32 articles in the review, we identified several themes, including philosophical underpinnings of fairness, IF metrics, mitigation methods for achieving IF, implications of achieving IF on group fairness and vice versa, and applications of IF in healthcare.</p><p><strong>Discussion: </strong>We find that research of IF is still in their early stages, particularly in healthcare, as evidenced by the limited number of relevant articles published between 2013 and 2024. While healthcare applications of IF remain sparse, growth has been steady in number of publications since 2012. The limitations of group fairness further emphasize the need for alternative approaches like IF. However, IF itself is not without challenges, including subjective definitions of similarity and potential bias encoding from data-driven methods. These findings, coupled with the limitations of the review process, underscore the need for more comprehensive research on the evolution of IF metrics and definitions to advance this promising field.</p><p><strong>Conclusion: </strong>While significant work has been done on algorithmic IF in recent years, the definition, use, and study of IF remain in their infancy, especially in healthcare. Future research is needed to comprehensively apply and evaluate IF in healthcare.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"8 1","pages":"ooae149"},"PeriodicalIF":2.5,"publicationDate":"2024-12-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11684587/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142907645","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"<i>pyBioPortal</i>: a Python package for simplifying cBioPortal data access in cancer research.","authors":"Matteo Valerio, Alessandro Inno, Stefania Gori","doi":"10.1093/jamiaopen/ooae146","DOIUrl":"10.1093/jamiaopen/ooae146","url":null,"abstract":"<p><strong>Objectives: </strong>In recent years, the rise of big data and artificial intelligence has led to an increasing expansion of databases and web services in biomedical research. cBioPortal is one of the most widely used platforms for accessing cancer genomic and clinical data. The primary objective of this study was to develop a tool that simplifies programmatic interaction with cBioPortal's web service.</p><p><strong>Materials and methods: </strong>We developed the <i>pyBioPortal</i> Python package, which leverages the cBioPortal REST API to access genomic and clinical data. The retrieved data is returned as a Pandas DataFrame, a format widely used for data analysis in Python.</p><p><strong>Results: </strong><i>pyBioPortal</i> offers an efficient interface between the user and the cBioPortal database. The data is provided in formats conducive to further analysis and visualization, promoting workflows and improving reproducibility.</p><p><strong>Discussion: </strong>The development of <i>pyBioPortal</i> addresses the challenge of accessing and processing large volumes of biomedical data. By simplifying the interaction with the cBioPortal API and providing data in Pandas DataFrame format, <i>pyBioPortal</i> allows users to focus more on the analytical aspects rather than data extraction.</p><p><strong>Conclusion: </strong>This tool facilitates the retrieval of heterogeneous biological and clinical data in a standardized format, making it more accessible for analysis and enhancing the reproducibility of results in cancer informatics. Distributed as an open-source project, <i>pyBioPortal</i> is available to the broader bioinformatics community, promoting collaboration and advancing research in cancer genomics.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"8 1","pages":"ooae146"},"PeriodicalIF":2.5,"publicationDate":"2024-12-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11671144/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142903754","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"xMEN: a modular toolkit for cross-lingual medical entity normalization.","authors":"Florian Borchert, Ignacio Llorca, Roland Roller, Bert Arnrich, Matthieu-P Schapranow","doi":"10.1093/jamiaopen/ooae147","DOIUrl":"10.1093/jamiaopen/ooae147","url":null,"abstract":"<p><strong>Objective: </strong>To improve performance of medical entity normalization across many languages, especially when fewer language resources are available compared to English.</p><p><strong>Materials and methods: </strong>We propose xMEN, a modular system for cross-lingual (x) medical entity normalization (MEN), accommodating both low- and high-resource scenarios. To account for the scarcity of aliases for many target languages and terminologies, we leverage multilingual aliases via cross-lingual candidate generation. For candidate ranking, we incorporate a trainable cross-encoder (CE) model if annotations for the target task are available. To balance the output of general-purpose candidate generators with subsequent trainable re-rankers, we introduce a novel rank regularization term in the loss function for training CEs. For re-ranking without gold-standard annotations, we introduce multiple new weakly labeled datasets using machine translation and projection of annotations from a high-resource language.</p><p><strong>Results: </strong>xMEN improves the state-of-the-art performance across various benchmark datasets for several European languages. Weakly supervised CEs are effective when no training data is available for the target task.</p><p><strong>Discussion: </strong>We perform an analysis of normalization errors, revealing that complex entities are still challenging to normalize. New modules and benchmark datasets can be easily integrated in the future.</p><p><strong>Conclusion: </strong>xMEN exhibits strong performance for medical entity normalization in many languages, even when no labeled data and few terminology aliases for the target language are available. To enable reproducible benchmarks in the future, we make the system available as an open-source Python toolkit. The pre-trained models and source code are available online: https://github.com/hpi-dhc/xmen.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"8 1","pages":"ooae147"},"PeriodicalIF":2.5,"publicationDate":"2024-12-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11671143/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142903763","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Designing a blockchain technology platform for enhancing the pre-exposure prophylaxis care continuum.","authors":"Anjum Khurshid, Daniel Toshio Harrell, Dennis Li, Camden Hallmark, Ladd Hanson, Nishi Viswanathan, Michelle Carr, Armand Brown, Marlene McNeese, Kayo Fujimoto","doi":"10.1093/jamiaopen/ooae140","DOIUrl":"10.1093/jamiaopen/ooae140","url":null,"abstract":"<p><strong>Objectives: </strong>Pre-exposure prophylaxis (PrEP) is a key biomedical intervention for ending the HIV epidemic in the United States, but its uptake is impeded by systemic barriers, including fragmented workflows and ineffective data coordination. This study aims to design PrEPLinker, a blockchain-based, client-centered platform to enhance care to address these challenges by improving care coordination and enabling clients to securely manage their identity and PrEP-related data.</p><p><strong>Materials and methods: </strong>Using Houston, Texas, as a use case, we conducted a needs assessment with PrEP collaborators to evaluate existing workflows and identify barriers in the PrEP care continuum. Based on these findings, we designed PrEPinker, a blockchain-based identity framework and digital wallet using self-sovereign identity and verifiable credentials (VCs). These features enable clients to securirely control their identity data and facilitate efficient, privacy-serving data sharing across PrEP service points, such as community testing sites, clinics, and pharmacies.</p><p><strong>Results: </strong>The needs assessment identified significant gaps in data exchange for PrEP referrals and follow-up appointments. In response, PrEPLinker was designed to incorporate decentralized identifiers-unique, secure digital identifiers that are not linked to any centralized authority-and VCs for ensuring seamless transfer of digital medical records. Preliminary usability testing with 15 participants showed that over 70% rated the interactive design positively, finding it easy to use, learn, and navigate without technical support. Additionally, more than 80% expressed confidence in using the blockchain based platform to manage sensitive health information securely.</p><p><strong>Discussion and conclusion: </strong>Blockchain technology offers a promising, client-centered solution for addressing systemic barriers in PrEP care by improving data cordination, security, and client control over personal health information. The design of PrEPLinker demorates the potential to streamline PrEP referrals, follow-up processes, and data managent. These advancements in data coordination and secruity could improve PrEP uptake and adherence, supporting efforts to reduce HIV transmission in Houston and beyond.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"7 4","pages":"ooae140"},"PeriodicalIF":2.5,"publicationDate":"2024-12-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11658693/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142865579","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"The FAIR database: facilitating access to public health research literature.","authors":"Zhixue Zhao, James Thomas, Gregory Kell, Claire Stansfield, Mark Clowes, Sergio Graziosi, Jeff Brunton, Iain James Marshall, Mark Stevenson","doi":"10.1093/jamiaopen/ooae139","DOIUrl":"10.1093/jamiaopen/ooae139","url":null,"abstract":"<p><strong>Objectives: </strong>In public health, access to research literature is critical to informing decision-making and to identify knowledge gaps. However, identifying relevant research is not a straightforward task since public health interventions are often complex, can have positive and negative impacts on health inequalities and are applied in diverse and rapidly evolving settings. We developed a \"living\" database of public health research literature to facilitate access to this information using Natural Language Processing tools.</p><p><strong>Materials and methods: </strong>Classifiers were identified to identify the study design (eg, cohort study or clinical trial) and relationship to factors that may be relevant to inequalities using the PROGRESS-Plus classification scheme. Training data were obtained from existing MEDLINE labels and from a set of systematic reviews in which studies were annotated with PROGRESS-Plus categories.</p><p><strong>Results: </strong>Evaluation of the classifiers showed that the study type classifier achieved average precision and recall of 0.803 and 0.930, respectively. The PROGRESS-Plus classification proved more challenging with average precision and recall of 0.608 and 0.534. The FAIR database uses information provided by these classifiers to facilitate access to inequality-related public health literature.</p><p><strong>Discussion: </strong>Previous work on automation of evidence synthesis has focused on clinical areas rather than public health, despite the need being arguably greater.</p><p><strong>Conclusion: </strong>The development of the FAIR database demonstrates that it is possible to create a publicly accessible and regularly updated database of public health research literature focused on inequalities. The database is freely available from https://eppi.ioe.ac.uk/eppi-vis/Fair.</p><p><strong>Netscc id number: </strong>NIHR133603.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"7 4","pages":"ooae139"},"PeriodicalIF":2.5,"publicationDate":"2024-12-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11641844/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142830099","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A dynamic customized electronic health record rule based clinical decision support tool for standardized adult intensive care metrics.","authors":"Eric W Cucchi, Joseph Burzynski, Nicholas Marshall, Bruce Greenberg","doi":"10.1093/jamiaopen/ooae143","DOIUrl":"10.1093/jamiaopen/ooae143","url":null,"abstract":"<p><strong>Objectives: </strong>Many routine patient care items should be reviewed at least daily for intensive care unit (ICU) patients. These items are often incompletely performed, and dynamic clinical decision support tools (CDSTs) may improve attention to these daily items. We sought to evaluate the accuracy of institutionalized electronic health record (EHR) based custom dynamic CDST to support 22 ICU rounding quality metrics across 7 categories (hypoglycemia, venothromboembolism prophylaxis, stress ulcer prophylaxis, mechanical ventilation, sedation, nutrition, and catheter removal).</p><p><strong>Design: </strong>The dynamic CDST evaluates patient characteristics and patient orders, then identifies gaps between active interventions and conditions with recommendations of evidence based clinical practice guidelines across 22 areas of care for each patient. The results of the tool prompt clinicians to address any identified care gaps. We completed a confusion matrix to assess the sensitivity, specificity, accuracy, positive predictive value (PPV), and negative predictive value (NPV) of the dynamic CDST and the individual metrics.</p><p><strong>Setting: </strong>Tertiary academic medical center and community hospital ICUs.</p><p><strong>Subject: </strong>Customized Clinical Decision Support Tool.</p><p><strong>Measurements and main results: </strong>The metrics were evaluated 1421 times over 484 patients. The overall accuracy of the entire dynamic CDST is 0.979 with a sensitivity of 0.979, specificity of 0.978, PPV 0.969, and NPV 0.986.</p><p><strong>Conclusions: </strong>A customized, EHR based dynamic CDST can be highly accurate. Integrating a comprehensive dynamic CDST into existing workflows could improve attention and actions related to routine ICU quality metrics.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"7 4","pages":"ooae143"},"PeriodicalIF":2.5,"publicationDate":"2024-12-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11633943/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142814357","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Development, deployment, and continuous monitoring of a machine learning model to predict respiratory failure in critically ill patients.","authors":"Jonathan Y Lam, Xiaolei Lu, Supreeth P Shashikumar, Ye Sel Lee, Michael Miller, Hayden Pour, Aaron E Boussina, Alex K Pearce, Atul Malhotra, Shamim Nemati","doi":"10.1093/jamiaopen/ooae141","DOIUrl":"10.1093/jamiaopen/ooae141","url":null,"abstract":"<p><strong>Objectives: </strong>This study describes the development and deployment of a machine learning (ML) model called Vent.io to predict mechanical ventilation (MV).</p><p><strong>Materials and methods: </strong>We trained Vent.io using electronic health record data of adult patients admitted to the intensive care units (ICUs) of the University of California San Diego (UCSD) Health System. We prospectively deployed Vent.io using a real-time platform at UCSD and evaluated the performance of Vent.io for a 1-month period in silent mode and on the MIMIC-IV dataset. As part of deployment, we included a Predetermined Changed Control Plan (PCCP) for continuous model monitoring that triggers model fine-tuning if performance drops below a specified area under the receiver operating curve (AUC) threshold of 0.85.</p><p><strong>Results: </strong>The Vent.io model had a median AUC of 0.897 (IQR: 0.892-0.904) with specificity of 0.81 (IQR: 0.812-0.841) and positive predictive value (PPV) of 0.174 (IQR: 0.148-0.176) at a fixed sensitivity of 0.6 during 10-fold cross validation and an AUC of 0.908, sensitivity of 0.632, specificity of 0.849, and PPV of 0.235 during prospective deployment. Vent.io had an AUC of 0.73 on the MIMIC-IV dataset, triggering model fine-tuning per the PCCP as the AUC was below the minimum of 0.85. The fine-tuned Vent.io model achieved an AUC of 0.873.</p><p><strong>Discussion: </strong>Deterioration of model performance is a significant challenge when deploying ML models prospectively or at different sites. Implementation of a PCCP can help models adapt to new patterns in data and maintain generalizability.</p><p><strong>Conclusion: </strong>Vent.io is a generalizable ML model that has the potential to improve patient care and resource allocation for ICU patients with need for MV.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"7 4","pages":"ooae141"},"PeriodicalIF":2.5,"publicationDate":"2024-12-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11633942/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142814439","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Integrated electronic health record tools to access real-world data in oncology research.","authors":"Michelle Casagni, Nicole Llewellyn, Maeve Kokolus, Miranda Chan, Robert Dingwell, Selina Chow, Nancy Campbell, Cassandra Elrahi, Steven Piantadosi, Andre Quina","doi":"10.1093/jamiaopen/ooae144","DOIUrl":"10.1093/jamiaopen/ooae144","url":null,"abstract":"<p><strong>Objectives: </strong>The Integrating Clinical Trials and Real-World Endpoints (ICAREdata) project aimed to demonstrate that electronic health record (EHR) data, expressed in a standard structured format, can be extracted and transmitted to contribute to clinical research. Using the minimal Common Oncology Data Elements (mCODE), we collected standardized oncology outcome data from EHRs across 10 clinical sites and 15 trials. This report details and assesses the ICAREdata technical implementation and offers recommendations to benefit future projects with similar goals.</p><p><strong>Materials and methods: </strong>In the ICAREdata project, we implemented tools to collect structured clinical outcome data within EHRs, then extract and transmit the mCODE-formatted data to a secure cloud storage platform. Using the socio-technical model of health information technology, we systematically assessed the technical implementations in this multi-institutional project.</p><p><strong>Results: </strong>We evaluated the ICAREdata method across the 8 inter-related dimensions of the socio-technical model. For each dimension, we identified challenges and developed recommendations that implementers of future initiatives can leverage.</p><p><strong>Discussion: </strong>The ICAREdata project successfully demonstrated the feasibility of using data standards for structured data capture and transmission in clinical trials. The lessons learned from this project can accelerate the success of similar initiatives using standards-based real-world data (RWD) capture and transmission for research.</p><p><strong>Conclusion: </strong>The ICAREdata project represents a step towards a future where researchers can access high-quality, standardized RWD leading to advances in research and improved care delivery.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"7 4","pages":"ooae144"},"PeriodicalIF":2.5,"publicationDate":"2024-12-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11633941/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142814446","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Investigation on the preferences for data quality assessment indicators of electronic health records: user-oriented perspective.","authors":"Liu Yang, Mudan Ren, Shuifa Sun, Ji Lu, Yirong Wu","doi":"10.1093/jamiaopen/ooae142","DOIUrl":"10.1093/jamiaopen/ooae142","url":null,"abstract":"<p><strong>Objectives: </strong>This study aims to investigate whether different types of electronic health record (EHR) users have distinct preferences for data quality assessment indicators (DQAI) and explore how these preferences can guide the enhancement of EHR systems and the optimization of related policies.</p><p><strong>Materials and methods: </strong>High-frequency indicators were identified by a systematic literature review to construct a DQAI system, which was assessed by a user-oriented investigation involving doctors, nurses, hospital supervisors, and clinical researchers. The entropy weight method and fuzzy comprehensive evaluation model were employed for the system comprehensive evaluation. Exploratory factor analysis was used to construct dimensions, and visualization analysis was utilized to explore preferences at both the indicator and dimension levels.</p><p><strong>Results: </strong>Sixteen indicators were identified to construct the DQAI system and grouped into 2 dimensions: structural and relational. The DQAI system achieved a comprehensive evaluation score of 90.445, corresponding to a \"very important\" membership level (62.5%). Doctors and nurses exhibited a higher score mean (4.43-4.66 out of 5) than supervisors (3.73-4.55 out of 5). Researchers emphasized credibility, with a score mean of 4.79 out of 5.</p><p><strong>Discussion: </strong>The findings reveal that different types of EHR users exhibit distinct preferences for the DQAI at both indicator and dimension levels. Doctors and nurses thought that all indicators were important, clinical researchers emphasized credibility, and supervisors focused mainly on accuracy. Indicators in the relational dimension were generally more valued than structural ones. Doctors and nurses prioritized indicators of relational dimension, while researchers and supervisors leaned towards indicators of structural dimension. These insights suggest that tailored approaches in EHR system development and policy-making could enhance EHR data quality.</p><p><strong>Conclusion: </strong>This study underscores the importance of user-centered approaches in optimizing EHR systems, highlighting diverse user preferences at both indicator and dimension levels.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"7 4","pages":"ooae142"},"PeriodicalIF":2.5,"publicationDate":"2024-12-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11633948/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142814452","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Decoding disparities: evaluating automatic speech recognition system performance in transcribing Black and White patient verbal communication with nurses in home healthcare.","authors":"Maryam Zolnoori, Sasha Vergez, Zidu Xu, Elyas Esmaeili, Ali Zolnour, Krystal Anne Briggs, Jihye Kim Scroggins, Seyed Farid Hosseini Ebrahimabad, James M Noble, Maxim Topaz, Suzanne Bakken, Kathryn H Bowles, Ian Spens, Nicole Onorato, Sridevi Sridharan, Margaret V McDonald","doi":"10.1093/jamiaopen/ooae130","DOIUrl":"10.1093/jamiaopen/ooae130","url":null,"abstract":"<p><strong>Objectives: </strong>As artificial intelligence evolves, integrating speech processing into home healthcare (HHC) workflows is increasingly feasible. Audio-recorded communications enhance risk identification models, with automatic speech recognition (ASR) systems as a key component. This study evaluates the transcription accuracy and equity of 4 ASR systems-Amazon Web Services (AWS) General, AWS Medical, Whisper, and Wave2Vec-in transcribing patient-nurse communication in US HHC, focusing on their ability in accurate transcription of speech from Black and White English-speaking patients.</p><p><strong>Materials and methods: </strong>We analyzed audio recordings of patient-nurse encounters from 35 patients (16 Black and 19 White) in a New York City-based HHC service. Overall, 860 utterances were available for study, including 475 drawn from Black patients and 385 from White patients. Automatic speech recognition performance was measured using word error rate (WER), benchmarked against a manual gold standard. Disparities were assessed by comparing ASR performance across racial groups using the linguistic inquiry and word count (LIWC) tool, focusing on 10 linguistic dimensions, as well as specific speech elements including repetition, filler words, and proper nouns (medical and nonmedical terms).</p><p><strong>Results: </strong>The average age of participants was 67.8 years (SD = 14.4). Communication lasted an average of 15 minutes (range: 11-21 minutes) with a median of 1186 words per patient. Of 860 total utterances, 475 were from Black patients and 385 from White patients. Amazon Web Services General had the highest accuracy, with a median WER of 39%. However, all systems showed reduced accuracy for Black patients, with significant discrepancies in LIWC dimensions such as \"Affect,\" \"Social,\" and \"Drives.\" Amazon Web Services Medical performed best for medical terms, though all systems have difficulties with filler words, repetition, and nonmedical terms, with AWS General showing the lowest error rates at 65%, 64%, and 53%, respectively.</p><p><strong>Discussion: </strong>While AWS systems demonstrated superior accuracy, significant disparities by race highlight the need for more diverse training datasets and improved dialect sensitivity. Addressing these disparities is critical for ensuring equitable ASR performance in HHC settings and enhancing risk prediction models through audio-recorded communication.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"7 4","pages":"ooae130"},"PeriodicalIF":2.5,"publicationDate":"2024-12-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11631515/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142808080","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}