Applied Clinical Informatics最新文献

{"title":"Evaluation of a Digital Scribe: Conversation Summarization for Emergency Department Consultation Calls.","authors":"Emre Sezgin, Joseph Winstead Sirrianni, Kelly Kranz","doi":"10.1055/a-2327-4121","DOIUrl":"10.1055/a-2327-4121","url":null,"abstract":"<p><strong>Objective: </strong>We present a proof-of-concept digital scribe system as an Emergency Department (ED) consultation call-based clinical conversation summarization pipeline to support clinical documentation, and report its performance.</p><p><strong>Materials and methods: </strong>We use four pre-trained large language models to establish the digital scribe system: T5-small, T5-base, PEGASUS-PubMed, and BART-Large-CNN via zero-shot and fine-tuning approaches. Our dataset includes 100 referral conversations among ED clinicians and medical records. We report the ROUGE-1, ROUGE-2, and ROUGE-L to compare model performance. In addition, we annotated transcriptions to assess the quality of generated summaries.</p><p><strong>Results: </strong>The fine-tuned BART-Large-CNN model demonstrates greater performance in summarization tasks with the highest ROUGE scores (F1ROUGE-1=0.49, F1ROUGE-2=0.23, F1ROUGE-L=0.35) scores. In contrast, PEGASUS-PubMed lags notably (F1ROUGE-1=0.28, F1ROUGE-2=0.11, F1ROUGE-L=0.22). BART-Large-CNN's performance decreases by more than 50% with the zero-shot approach. Annotations show that BART-Large-CNN performs 71.4% recall in identifying key information and a 67.7% accuracy rate.</p><p><strong>Discussion: </strong>The BART-Large-CNN model demonstrates a high level of understanding of clinical dialogue structure, indicated by its performance with and without fine-tuning. Despite some instances of high recall, there is variability in the model's performance, particularly in achieving consistent correctness, suggesting room for refinement. The model's recall ability varies across different information categories.</p><p><strong>Conclusion: </strong>The study provides evidence towards the potential of AI-assisted tools in assisting clinical documentation. Future work is suggested on expanding the research scope with additional language models and hybrid approaches, and comparative analysis to measure documentation burden and human factors.</p>","PeriodicalId":48956,"journal":{"name":"Applied Clinical Informatics","volume":null,"pages":null},"PeriodicalIF":2.1,"publicationDate":"2024-05-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11268986/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140946262","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Predicting Provider Workload Using Predicted Patient Risk Score and Social Determinants of Health in Primary Care Setting.","authors":"Yiqun Jiang, Yu-Li Huang, Alexandra Watral, Renaldo C Blocker, David R Rushlow","doi":"10.1055/s-0044-1787647","DOIUrl":"10.1055/s-0044-1787647","url":null,"abstract":"<p><strong>Background: </strong> Provider burnout due to workload is a significant concern in primary care settings. Workload for primary care providers encompasses both scheduled visit care and non-visit care interactions. These interactions are highly influenced by patients' health conditions or acuity, which can be measured by the Adjusted Clinical Group (ACG) score. However, new patients typically have minimal health information beyond social determinants of health (SDOH) to determine ACG score.</p><p><strong>Objectives: </strong> This study aims to assess new patient workload by first predicting the ACG score using SDOH, age, and gender and then using this information to estimate the number of appointments (scheduled visit care) and non-visit care interactions.</p><p><strong>Methods: </strong> Two years of appointment data were collected for patients who had initial appointment requests in the first year and had the ACG score, appointment, and non-visit care counts in the subsequent year. State-of-art machine learning algorithms were employed to predict ACG scores and compared with current baseline estimation. Linear regression models were then used to predict appointments and non-visit care interactions, integrating demographic data, SDOH, and predicted ACG scores.</p><p><strong>Results: </strong> The machine learning methods showed promising results in predicting ACG scores. Besides the decision tree, all other methods performed at least 9% better in accuracy than the baseline approach which had an accuracy of 78%. Incorporating SDOH and predicted ACG scores also significantly improved the prediction for both appointments and non-visit care interactions. The <i>R</i> ² values increased by 95.2 and 93.8%, respectively. Furthermore, age, smoking tobacco, family history, gender, usage of injection birth control, and ACG were significant factors for determining appointments. SDOH factors such as tobacco usage, physical exercise, education level, and group activities were strongly correlated with non-visit care interactions.</p><p><strong>Conclusion: </strong> The study highlights the importance of SDOH and predicted ACG scores in predicting provider workload in primary care settings.</p>","PeriodicalId":48956,"journal":{"name":"Applied Clinical Informatics","volume":null,"pages":null},"PeriodicalIF":2.1,"publicationDate":"2024-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11221991/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141499394","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"SALUS-A Study on Self-Tonometry for Glaucoma Patients: Design and Implementation of the Electronic Case File.","authors":"Sandra Geisler, Kristina Oldiges, Florim Hamiti, Jens J Storp, M A Masud, Julian A Zimmermann, Stefan Kreutter, Nicole Eter, Thomas Berlage","doi":"10.1055/s-0044-1787008","DOIUrl":"10.1055/s-0044-1787008","url":null,"abstract":"<p><strong>Background: </strong> In times of omnipresent digitization and big data, telemedicine and electronic case files (ECFs) are gaining ground for networking between players in the health care sector. In the context of the SALUS study, this approach is applied in practice in the form of electronic platforms to display and process disease-relevant data of glaucoma patients.</p><p><strong>Objectives: </strong> The SALUS ECF is designed and implemented to support data acquisition and presentation, monitoring, and outcome control for patients suffering from glaucoma in a clinical setting. Its main aim is to provide a means for out- and inpatient exchange of information between various stakeholders with an intuitive user interface in ophthalmologic care. Instrument data, anamnestic data, and diagnostic assessments need to be accessible and historic data stored for patient monitoring. Quality control of the data is ensured by a reading center.</p><p><strong>Methods: </strong> Based on an intensive requirement analysis, we implemented the ECF as a web-based application in React with a Datomic back-end exposing REST and GraphQL APIs for data access and import. A flexible role management was developed, which addresses the various tasks of multiple stakeholders in the SALUS study. Data security is ensured by a comprehensive encryption concept. We evaluated the usability and efficiency of the ECF by measuring the durations medical doctors need to enter and work with the data.</p><p><strong>Results: </strong> The evaluation showed that the ECF is time-saving in comparison to paper-based assessments and offers supportive monitoring and outcome control for numerical and imaging-related data. By allowing patients and physicians to access the digital ECF, data connectivity as well as patient autonomy were enhanced.</p><p><strong>Conclusion: </strong> ECFs have a great potential to efficiently support all patients and stakeholders involved in the care of glaucoma patients. They benefit from the efficient management and view of the data tailored to their specific role.</p>","PeriodicalId":48956,"journal":{"name":"Applied Clinical Informatics","volume":null,"pages":null},"PeriodicalIF":2.1,"publicationDate":"2024-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11186700/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141428048","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Contributors to Electronic Health Record-Integrated Secure Messaging Use: A Study of Over 33,000 Health Care Professionals.","authors":"Laura R Baratta, Daphne Lew, Thomas Kannampallil, Sunny S Lou","doi":"10.1055/s-0044-1787756","DOIUrl":"10.1055/s-0044-1787756","url":null,"abstract":"<p><strong>Objectives: </strong> Electronic health record (EHR)-integrated secure messaging is extensively used for communication between clinicians. We investigated the factors contributing to secure messaging use in a large health care system.</p><p><strong>Methods: </strong> This was a cross-sectional study that included 14 hospitals and 263 outpatient clinic locations. Data on EHR-integrated secure messaging use over a 1-month period (February 1, 2023, through February 28, 2023) were collected. A multilevel mixed effects model was used to assess the contribution of clinical role, clinical unit (i.e., specific inpatient ward or outpatient clinic), hospital or clinic location (i.e., Hospital X or Outpatient Clinic Building Y), and inpatient versus outpatient setting toward secure messaging use.</p><p><strong>Results: </strong> Of the 33,195 health care professionals who worked during the study period, 20,576 (62%) were secure messaging users. In total, 25.3% of the variability in messaging use was attributable to the clinical unit and 30.5% was attributable to the hospital or clinic location. Compared with nurses, advanced practice providers, pharmacists, and physicians were more likely to use secure messaging, whereas medical assistants, social workers, and therapists were less likely (<i>p</i> < 0.001). After adjusting for other factors, inpatient versus outpatient setting was not associated with secure messaging use.</p><p><strong>Conclusion: </strong> Secure messaging was widely used; however, there was substantial variation by clinical role, clinical unit, and hospital or clinic location. Our results suggest that interventions and policies for managing secure messaging behaviors are likely to be most effective if they are not only set at the organizational level but also communicated and tailored toward individual clinical units and clinician workflows.</p>","PeriodicalId":48956,"journal":{"name":"Applied Clinical Informatics","volume":null,"pages":null},"PeriodicalIF":2.1,"publicationDate":"2024-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11268985/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141761946","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A Neurosurgical Readmissions Reduction Program in an Academic Hospital Leveraging Machine Learning, Workflow Analysis, and Simulation.","authors":"Tzu-Chun Wu, Abraham Kim, Ching-Tzu Tsai, Andy Gao, Taran Ghuman, Anne Paul, Alexandra Castillo, Joseph Cheng, Owoicho Adogwa, Laura B Ngwenya, Brandon Foreman, Danny T Y Wu","doi":"10.1055/s-0044-1787119","DOIUrl":"10.1055/s-0044-1787119","url":null,"abstract":"<p><strong>Background: </strong> Predicting 30-day hospital readmissions is crucial for improving patient outcomes, optimizing resource allocation, and achieving financial savings. Existing studies reporting the development of machine learning (ML) models predictive of neurosurgical readmissions do not report factors related to clinical implementation.</p><p><strong>Objectives: </strong> Train individual predictive models with good performance (area under the receiver operating characteristic curve or AUROC > 0.8), identify potential interventions through semi-structured interviews, and demonstrate estimated clinical and financial impact of these models.</p><p><strong>Methods: </strong> Electronic health records were utilized with five ML methodologies: gradient boosting, decision tree, random forest, ridge logistic regression, and linear support vector machine. Variables of interest were determined by domain experts and literature. The dataset was split divided 80% for training and validation and 20% for testing randomly. Clinical workflow analysis was conducted using semi-structured interviews to identify possible intervention points. Calibrated agent-based models (ABMs), based on a previous study with interventions, were applied to simulate reductions of the 30-day readmission rate and financial costs.</p><p><strong>Results: </strong> The dataset covered 12,334 neurosurgical intensive care unit (NSICU) admissions (11,029 patients); 1,903 spine surgery admissions (1,641 patients), and 2,208 traumatic brain injury (TBI) admissions (2,185 patients), with readmission rate of 13.13, 13.93, and 23.73%, respectively. The random forest model for NSICU achieved best performance with an AUROC score of 0.89, capturing potential patients effectively. Six interventions were identified through 12 semi-structured interviews targeting preoperative, inpatient stay, discharge phases, and follow-up phases. Calibrated ABMs simulated median readmission reduction rates and resulted in 13.13 to 10.12% (NSICU), 13.90 to 10.98% (spine surgery), and 23.64 to 21.20% (TBI). Approximately $1,300,614.28 in saving resulted from potential interventions.</p><p><strong>Conclusion: </strong> This study reports the successful development and simulation of an ML-based approach for predicting and reducing 30-day hospital readmissions in neurosurgery. The intervention shows feasibility in improving patient outcomes and reducing financial losses.</p>","PeriodicalId":48956,"journal":{"name":"Applied Clinical Informatics","volume":null,"pages":null},"PeriodicalIF":2.1,"publicationDate":"2024-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11186699/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141428047","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Human-Centered Design and Development of a Fall Prevention Exercise App for Older Adults in Primary Care Settings.","authors":"Nichole K Czuber, Pamela M Garabedian, Hannah Rice, Christian J Tejeda, Patricia C Dykes, Nancy K Latham","doi":"10.1055/a-2267-1727","DOIUrl":"10.1055/a-2267-1727","url":null,"abstract":"<p><strong>Background: </strong> Falls in older adults are a serious public health problem that can lead to reduced quality of life or death. Patients often do not receive fall prevention guidance from primary care providers (PCPs), despite evidence that falls can be prevented. Mobile health technologies may help to address this disparity and promote evidence-based fall prevention.</p><p><strong>Objective: </strong> Our main objective was to use human-centered design to develop a user-friendly, fall prevention exercise app using validated user requirements. The app features evidence-based behavior change strategies and exercise content to support older people initiating and adhering to a progressive fall prevention exercise program.</p><p><strong>Methods: </strong> We organized our multistage, iterative design process into three phases: gathering user requirements, usability evaluation, and refining app features. Our methods include focus groups, usability testing, and subject-matter expert meetings.</p><p><strong>Results: </strong> Focus groups (total <i>n</i> = 6), usability testing (<i>n</i> = 30) including a posttest questionnaire [Health-ITUES score: mean (standard deviation [SD]) = 4.2 (0.9)], and subject-matter expert meetings demonstrate participant satisfaction with the app concept and design. Overall, participants saw value in receiving exercise prescriptions from the app that would be recommended by their PCP and reported satisfaction with the content of the app.</p><p><strong>Conclusion: </strong> This study demonstrates the development, refinement, and usability testing of a fall prevention exercise app and corresponding tools that PCPs may use to prescribe tailored exercise recommendations to their older patients as an evidence-based fall prevention strategy accessible in the context of busy clinical workflows.</p>","PeriodicalId":48956,"journal":{"name":"Applied Clinical Informatics","volume":null,"pages":null},"PeriodicalIF":2.1,"publicationDate":"2024-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11236445/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139730802","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Implementation of a Real-Time Documentation Assistance Tool: Automated Diagnosis (AutoDx).","authors":"Matthew T Cerasale, Ali Mansour, Ethan Molitch-Hou, Sean Bernstein, Tokhanh Nguyen, Cheng-Kai Kao","doi":"10.1055/a-2319-0598","DOIUrl":"10.1055/a-2319-0598","url":null,"abstract":"<p><strong>Background: </strong> Clinical documentation improvement programs are utilized by most health care systems to enhance provider documentation. Suggestions are sent to providers in a variety of ways, and are commonly referred to as coding queries. Responding to these coding queries can require significant provider time and do not often align with workflows. To enhance provider documentation in a more consistent manner without creating undue burden, alternative strategies are required.</p><p><strong>Objectives: </strong> The aim of this study is to evaluate the impact of a real-time documentation assistance tool, named AutoDx, on the volume of coding queries and encounter-level outcome metrics, including case-mix index (CMI).</p><p><strong>Methods: </strong> The AutoDx tool was developed utilizing tools existing within the electronic health record, and is based on the generation of messages when clinical conditions are met. These messages appear within provider notes and required little to no interaction. Initial diagnoses included in the tool were electrolyte deficiencies, obesity, and malnutrition. The tool was piloted in a cohort of Hospital Medicine providers, then expanded to the Neuro Intensive Care Unit (NICU), with addition diagnoses being added.</p><p><strong>Results: </strong> The initial Hospital Medicine implementation evaluation included 590 encounters pre- and 531 post-implementation. The volume of coding queries decreased 57% (<i>p</i> < 0.0001) for the targeted diagnoses compared with 6% (<i>p</i> = 0.77) in other high-volume diagnoses. In the NICU cohort, 829 encounters pre-implementation were compared with 680 post. The proportion of AutoDx coding queries compared with all other coding queries decreased from 54.9 to 37.1% (<i>p</i> < 0.0001). During the same period, CMI demonstrated a significant increase post-implementation (4.00 vs. 4.55, <i>p</i> = 0.02).</p><p><strong>Conclusion: </strong> The real-time documentation assistance tool led to a significant decrease in coding queries for targeted diagnoses in two unique provider cohorts. This improvement was also associated with a significant increase in CMI during the implementation time period.</p>","PeriodicalId":48956,"journal":{"name":"Applied Clinical Informatics","volume":null,"pages":null},"PeriodicalIF":2.1,"publicationDate":"2024-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11208109/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140860521","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Comparing Clinician Estimates versus a Statistical Tool for Predicting Risk of Death within 45 Days of Admission for Cancer Patients.","authors":"Adrianna Z Herskovits, Tiffanny Newman, Kevin Nicholas, Cesar F Colorado-Jimenez, Claire E Perry, Alisa Valentino, Isaac Wagner, Barbara Egan, Dmitriy Gorenshteyn, Andrew J Vickers, Melissa S Pessin","doi":"10.1055/s-0044-1787185","DOIUrl":"10.1055/s-0044-1787185","url":null,"abstract":"<p><strong>Objectives: </strong> While clinical practice guidelines recommend that oncologists discuss goals of care with patients who have advanced cancer, it is estimated that less than 20% of individuals admitted to the hospital with high-risk cancers have end-of-life discussions with their providers. While there has been interest in developing models for mortality prediction to trigger such discussions, few studies have compared how such models compare with clinical judgment to determine a patient's mortality risk.</p><p><strong>Methods: </strong> This study is a prospective analysis of 1,069 solid tumor medical oncology hospital admissions (<i>n</i> = 911 unique patients) from February 7 to June 7, 2022, at Memorial Sloan Kettering Cancer Center. Electronic surveys were sent to hospitalists, advanced practice providers, and medical oncologists the first afternoon following a hospital admission and they were asked to estimate the probability that the patient would die within 45 days. Provider estimates of mortality were compared with those from a predictive model developed using a supervised machine learning methodology, and incorporated routine laboratory, demographic, biometric, and admission data. Area under the receiver operating characteristic curve (AUC), calibration and decision curves were compared between clinician estimates and the model predictions.</p><p><strong>Results: </strong> Within 45 days following hospital admission, 229 (25%) of 911 patients died. The model performed better than the clinician estimates (AUC 0.834 vs. 0.753, <i>p</i> < 0.0001). Integrating clinician predictions with the model's estimates further increased the AUC to 0.853 (<i>p</i> < 0.0001). Clinicians overestimated risk whereas the model was extremely well-calibrated. The model demonstrated net benefit over a wide range of threshold probabilities.</p><p><strong>Conclusion: </strong> The inpatient prognosis at admission model is a robust tool to assist clinical providers in evaluating mortality risk, and it has recently been implemented in the electronic medical record at our institution to improve end-of-life care planning for hospitalized cancer patients.</p>","PeriodicalId":48956,"journal":{"name":"Applied Clinical Informatics","volume":null,"pages":null},"PeriodicalIF":2.1,"publicationDate":"2024-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11208110/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141460184","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Social Media's Lessons for Clinical Decision Support: Strategies to Improve Engagement and Acceptance.","authors":"Christopher Sova, Eric Poon, Robert Clayton Musser, Anand Chowdhury","doi":"10.1055/s-0044-1787648","DOIUrl":"10.1055/s-0044-1787648","url":null,"abstract":"","PeriodicalId":48956,"journal":{"name":"Applied Clinical Informatics","volume":null,"pages":null},"PeriodicalIF":2.1,"publicationDate":"2024-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11221992/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141499395","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Manual Evaluation of Record Linkage Algorithm Performance in Four Real-World Datasets.","authors":"Agrayan K Gupta, Huiping Xu, Xiaochun Li, Joshua R Vest, Shaun J Grannis","doi":"10.1055/a-2291-1391","DOIUrl":"10.1055/a-2291-1391","url":null,"abstract":"<p><strong>Objectives: </strong> Patient data are fragmented across multiple repositories, yielding suboptimal and costly care. Record linkage algorithms are widely accepted solutions for improving completeness of patient records. However, studies often fail to fully describe their linkage techniques. Further, while many frameworks evaluate record linkage methods, few focus on producing gold standard datasets. This highlights a need to assess these frameworks and their real-world performance. We use real-world datasets and expand upon previous frameworks to evaluate a consistent approach to the manual review of gold standard datasets and measure its impact on algorithm performance.</p><p><strong>Methods: </strong> We applied the framework, which includes elements for data description, reviewer training and adjudication, and software and reviewer descriptions, to four datasets. Record pairs were formed and between 15,000 and 16,500 records were randomly sampled from these pairs. After training, two reviewers determined match status for each record pair. If reviewers disagreed, a third reviewer was used for final adjudication.</p><p><strong>Results: </strong> Between the four datasets, the percent discordant rate ranged from 1.8 to 13.6%. While reviewers' discordance rate typically ranged between 1 and 5%, one exhibited a 59% discordance rate, showing the importance of the third reviewer. The original analysis was compared with three sensitivity analyses. The original analysis most often exhibited the highest predictive values compared with the sensitivity analyses.</p><p><strong>Conclusion: </strong> Reviewers vary in their assessment of a gold standard, which can lead to variances in estimates for matching performance. Our analysis demonstrates how a multireviewer process can be applied to create gold standards, identify reviewer discrepancies, and evaluate algorithm performance.</p>","PeriodicalId":48956,"journal":{"name":"Applied Clinical Informatics","volume":null,"pages":null},"PeriodicalIF":2.1,"publicationDate":"2024-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11290950/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140177354","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}