JAMIA Open最新文献

{"title":"Development of REDCap-based architecture for a clinical decision support tool linked to the electronic health record for assessment of medication appropriateness.","authors":"Peter A Charpentier, Marcia C Mecca, Cynthia Brandt, Terri R Fried","doi":"10.1093/jamiaopen/ooad041","DOIUrl":"10.1093/jamiaopen/ooad041","url":null,"abstract":"<p><strong>Objective: </strong>To develop the architecture for a clinical decision support system (CDSS) linked to the electronic health record (EHR) using the tools provided by Research Electronic Data Capture (REDCap) to assess medication appropriateness in older adults with polypharmacy.</p><p><strong>Materials and methods: </strong>The tools available in REDCap were used to create the architecture for replicating a previously developed stand-alone system while overcoming its limitations.</p><p><strong>Results: </strong>The architecture consists of data input forms, drug- and disease-mapper, rules engine, and report generator. The input forms integrate medication and health condition data from the EHR with patient assessment data. The rules engine evaluates medication appropriateness through rules built through a series of drop-down menus. The rules generate output, which are a set of recommendations to the clinician.</p><p><strong>Discussion and conclusion: </strong>This architecture successfully replicates the stand-alone CDSS while addressing its limitations. It is compatible with several EHRs, easily shared among the large community using REDCap, and readily modifiable.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"6 2","pages":"ooad041"},"PeriodicalIF":2.1,"publicationDate":"2023-06-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/39/73/ooad041.PMC10276359.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"9662486","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Developing, pilot testing, and refining requirements for 3 EHR-integrated interventions to improve diagnostic safety in acute care: a user-centered approach.","authors":"Alison Garber, Pamela Garabedian, Lindsey Wu, Alyssa Lam, Maria Malik, Hannah Fraser, Kerrin Bersani, Nicholas Piniella, Daniel Motta-Calderon, Ronen Rozenblum, Kumiko Schnock, Jacqueline Griffin, Jeffrey L Schnipper, David W Bates, Anuj K Dalal","doi":"10.1093/jamiaopen/ooad031","DOIUrl":"10.1093/jamiaopen/ooad031","url":null,"abstract":"<p><strong>Objective: </strong>To describe a user-centered approach to develop, pilot test, and refine requirements for 3 electronic health record (EHR)-integrated interventions that target key diagnostic process failures in hospitalized patients.</p><p><strong>Materials and methods: </strong>Three interventions were prioritized for development: a Diagnostic Safety Column (<i>DSC</i>) within an EHR-integrated dashboard to identify at-risk patients; a Diagnostic Time-Out (<i>DTO</i>) for clinicians to reassess the working diagnosis; and a Patient Diagnosis Questionnaire (<i>PDQ</i>) to gather patient concerns about the diagnostic process. Initial requirements were refined from analysis of test cases with elevated risk predicted by <i>DSC</i> logic compared to risk perceived by a clinician working group; <i>DTO</i> testing sessions with clinicians; <i>PDQ</i> responses from patients; and focus groups with clinicians and patient advisors using storyboarding to model the integrated interventions. Mixed methods analysis of participant responses was used to identify final requirements and potential implementation barriers.</p><p><strong>Results: </strong>Final requirements from analysis of 10 test cases predicted by the <i>DSC</i>, 18 clinician <i>DTO</i> participants, and 39 <i>PDQ</i> responses included the following: <i>DSC</i> configurable parameters (variables, weights) to adjust baseline risk estimates in real-time based on new clinical data collected during hospitalization; more concise <i>DTO</i> wording and flexibility for clinicians to conduct the <i>DTO</i> with or without the patient present; and integration of <i>PDQ</i> responses into the <i>DSC</i> to ensure closed-looped communication with clinicians. Analysis of focus groups confirmed that tight integration of the interventions with the EHR would be necessary to prompt clinicians to reconsider the working diagnosis in cases with elevated diagnostic error (DE) risk or uncertainty. Potential implementation barriers included alert fatigue and distrust of the risk algorithm (<i>DSC</i>); time constraints, redundancies, and concerns about disclosing uncertainty to patients (<i>DTO</i>); and patient disagreement with the care team's diagnosis (<i>PDQ</i>).</p><p><strong>Discussion: </strong>A user-centered approach led to evolution of requirements for 3 interventions targeting key diagnostic process failures in hospitalized patients at risk for DE.</p><p><strong>Conclusions: </strong>We identify challenges and offer lessons from our user-centered design process.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"6 2","pages":"ooad031"},"PeriodicalIF":2.1,"publicationDate":"2023-05-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/a6/b8/ooad031.PMC10172040.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"9523709","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A metadata framework for computational phenotypes.","authors":"Matthew Spotnitz, Nripendra Acharya, James J Cimino, Shawn Murphy, Bahram Namjou, Nancy Crimmins, Theresa Walunas, Cong Liu, David Crosslin, Barbara Benoit, Elisabeth Rosenthal, Jennifer A Pacheco, Anna Ostropolets, Harry Reyes Nieva, Jason S Patterson, Lauren R Richter, Tiffany J Callahan, Ahmed Elhussein, Chao Pang, Krzysztof Kiryluk, Jordan Nestor, Atlas Khan, Sumit Mohan, Evan Minty, Wendy Chung, Wei-Qi Wei, Karthik Natarajan, Chunhua Weng","doi":"10.1093/jamiaopen/ooad032","DOIUrl":"10.1093/jamiaopen/ooad032","url":null,"abstract":"<p><p>With the burgeoning development of computational phenotypes, it is increasingly difficult to identify the right phenotype for the right tasks. This study uses a mixed-methods approach to develop and evaluate a novel metadata framework for retrieval of and reusing computational phenotypes. Twenty active phenotyping researchers from 2 large research networks, Electronic Medical Records and Genomics and Observational Health Data Sciences and Informatics, were recruited to suggest metadata elements. Once consensus was reached on 39 metadata elements, 47 new researchers were surveyed to evaluate the utility of the metadata framework. The survey consisted of 5-Likert multiple-choice questions and open-ended questions. Two more researchers were asked to use the metadata framework to annotate 8 type-2 diabetes mellitus phenotypes. More than 90% of the survey respondents rated metadata elements regarding phenotype definition and validation methods and metrics positively with a score of 4 or 5. Both researchers completed annotation of each phenotype within 60 min. Our thematic analysis of the narrative feedback indicates that the metadata framework was effective in capturing rich and explicit descriptions and enabling the search for phenotypes, compliance with data standards, and comprehensive validation metrics. Current limitations were its complexity for data collection and the entailed human costs.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"6 2","pages":"ooad032"},"PeriodicalIF":2.1,"publicationDate":"2023-05-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10168627/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"10537266","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A machine learning approach to determine resilience utilizing wearable device data: analysis of an observational cohort.","authors":"Robert P Hirten, Maria Suprun, Matteo Danieletto, Micol Zweig, Eddye Golden, Renata Pyzik, Sparshdeep Kaur, Drew Helmus, Anthony Biello, Kyle Landell, Jovita Rodrigues, Erwin P Bottinger, Laurie Keefer, Dennis Charney, Girish N Nadkarni, Mayte Suarez-Farinas, Zahi A Fayad","doi":"10.1093/jamiaopen/ooad029","DOIUrl":"10.1093/jamiaopen/ooad029","url":null,"abstract":"<p><strong>Objective: </strong>To assess whether an individual's degree of psychological resilience can be determined from physiological metrics passively collected from a wearable device.</p><p><strong>Materials and methods: </strong>Data were analyzed in this secondary analysis of the Warrior Watch Study dataset, a prospective cohort of healthcare workers enrolled across 7 hospitals in New York City. Subjects wore an Apple Watch for the duration of their participation. Surveys were collected measuring resilience, optimism, and emotional support at baseline.</p><p><strong>Results: </strong>We evaluated data from 329 subjects (mean age 37.4 years, 37.1% male). Across all testing sets, gradient-boosting machines (GBM) and extreme gradient-boosting models performed best for high- versus low-resilience prediction, stratified on a median Connor-Davidson Resilience Scale-2 score of 6 (interquartile range = 5-7), with an AUC of 0.60. When predicting resilience as a continuous variable, multivariate linear models had a correlation of 0.24 (<i>P</i> = .029) and RMSE of 1.37 in the testing data. A positive psychological construct, comprised of resilience, optimism, and emotional support was also evaluated. The oblique random forest method performed best in estimating high- versus low-composite scores stratified on a median of 32.5, with an AUC of 0.65, a sensitivity of 0.60, and a specificity of 0.70.</p><p><strong>Discussion: </strong>In a <i>post hoc</i> analysis, machine learning models applied to physiological metrics collected from wearable devices had some predictive ability in identifying resilience states and a positive psychological construct.</p><p><strong>Conclusions: </strong>These findings support the further assessment of psychological characteristics from passively collected wearable data in dedicated studies.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"6 2","pages":"ooad029"},"PeriodicalIF":2.5,"publicationDate":"2023-05-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/41/01/ooad029.PMC10152991.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"9415118","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"pyPheWAS Explorer: a visualization tool for exploratory analysis of phenome-disease associations.","authors":"Cailey I Kerley, Tin Q Nguyen, Karthik Ramadass, Laurie E Cutting, Bennett A Landman, Matthew Berger","doi":"10.1093/jamiaopen/ooad018","DOIUrl":"10.1093/jamiaopen/ooad018","url":null,"abstract":"<p><strong>Objective: </strong>To enable interactive visualization of phenome-wide association studies (PheWAS) on electronic health records (EHR).</p><p><strong>Materials and methods: </strong>Current PheWAS technologies require familiarity with command-line interfaces and lack end-to-end data visualizations. pyPheWAS Explorer allows users to examine group variables, test assumptions, design PheWAS models, and evaluate results in a streamlined graphical interface.</p><p><strong>Results: </strong>A cohort of attention deficit hyperactivity disorder (ADHD) subjects and matched non-ADHD controls is examined. pyPheWAS Explorer is used to build a PheWAS model including sex and deprivation index as covariates, and the Explorer's result visualization for this model reveals known ADHD comorbidities.</p><p><strong>Discussion: </strong>pyPheWAS Explorer may be used to rapidly investigate potentially novel EHR associations. Broader applications include deployment for clinical experts and preliminary exploration tools for institutional EHR repositories.</p><p><strong>Conclusion: </strong>pyPheWAS Explorer provides a seamless graphical interface for designing, executing, and analyzing PheWAS experiments, emphasizing exploratory analysis of regression types and covariate selection.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"6 1","pages":"ooad018"},"PeriodicalIF":2.1,"publicationDate":"2023-04-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10070037/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"9311923","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Accelerated curation of checkpoint inhibitor-induced colitis cases from electronic health records.","authors":"Protiva Rahman,&nbsp;Cheng Ye,&nbsp;Kathleen F Mittendorf,&nbsp;Michele Lenoue-Newton,&nbsp;Christine Micheel,&nbsp;Jan Wolber,&nbsp;Travis Osterman,&nbsp;Daniel Fabbri","doi":"10.1093/jamiaopen/ooad017","DOIUrl":"https://doi.org/10.1093/jamiaopen/ooad017","url":null,"abstract":"<p><strong>Objective: </strong>Automatically identifying patients at risk of immune checkpoint inhibitor (ICI)-induced colitis allows physicians to improve patientcare. However, predictive models require training data curated from electronic health records (EHR). Our objective is to automatically identify notes documenting ICI-colitis cases to accelerate data curation.</p><p><strong>Materials and methods: </strong>We present a data pipeline to automatically identify ICI-colitis from EHR notes, accelerating chart review. The pipeline relies on BERT, a state-of-the-art natural language processing (NLP) model. The first stage of the pipeline segments long notes using keywords identified through a logistic classifier and applies BERT to identify ICI-colitis notes. The next stage uses a second BERT model tuned to identify false positive notes and remove notes that were likely positive for mentioning colitis as a side-effect. The final stage further accelerates curation by highlighting the colitis-relevant portions of notes. Specifically, we use BERT's attention scores to find high-density regions describing colitis.</p><p><strong>Results: </strong>The overall pipeline identified colitis notes with 84% precision and reduced the curator note review load by 75%. The segment BERT classifier had a high recall of 0.98, which is crucial to identify the low incidence (<10%) of colitis.</p><p><strong>Discussion: </strong>Curation from EHR notes is a burdensome task, especially when the curation topic is complicated. Methods described in this work are not only useful for ICI colitis but can also be adapted for other domains.</p><p><strong>Conclusion: </strong>Our extraction pipeline reduces manual note review load and makes EHR data more accessible for research.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"6 1","pages":"ooad017"},"PeriodicalIF":2.1,"publicationDate":"2023-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/4f/2b/ooad017.PMC10066800.pdf","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"9616866","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"#Coronavirus on TikTok: user engagement with misinformation as a potential threat to public health behavior.","authors":"Jonathan D Baghdadi,&nbsp;K C Coffey,&nbsp;Rachael Belcher,&nbsp;James Frisbie,&nbsp;Naeemul Hassan,&nbsp;Danielle Sim,&nbsp;Rena D Malik","doi":"10.1093/jamiaopen/ooad013","DOIUrl":"https://doi.org/10.1093/jamiaopen/ooad013","url":null,"abstract":"<p><p>Coronavirus disease (COVID)-related misinformation is prevalent online, including on social media. The purpose of this study was to explore factors associated with user engagement with COVID-related misinformation on the social media platform, TikTok. A sample of TikTok videos associated with the hashtag #coronavirus was downloaded on September 20, 2020. Misinformation was evaluated on a scale (low, medium, and high) using a codebook developed by experts in infectious diseases. Multivariable modeling was used to evaluate factors associated with number of views and presence of user comments indicating intention to change behavior. One hundred and sixty-six TikTok videos were identified and reviewed. Moderate misinformation was present in 36 (22%) videos viewed a median of 6.8 million times (interquartile range [IQR] 3.6-16 million), and high-level misinformation was present in 11 (7%) videos viewed a median of 9.4 million times (IQR 5.1-18 million). After controlling for characteristics and content, videos containing moderate misinformation were less likely to generate a user response indicating intended behavior change. By contrast, videos containing high-level misinformation were less likely to be viewed but demonstrated a nonsignificant trend towards higher engagement among viewers. COVID-related misinformation is less frequently viewed on TikTok but more likely to engage viewers. Public health authorities can combat misinformation on social media by posting informative content of their own.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"6 1","pages":"ooad013"},"PeriodicalIF":2.1,"publicationDate":"2023-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9949581/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"9159279","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Problem-oriented documentation: design and widespread adoption of a novel toolkit in a commercial electronic health record.","authors":"Richard L Altman,&nbsp;Chen-Tan Lin,&nbsp;Mark Earnest","doi":"10.1093/jamiaopen/ooad005","DOIUrl":"https://doi.org/10.1093/jamiaopen/ooad005","url":null,"abstract":"<p><strong>Background: </strong>Problem-oriented documentation is an accepted method of note construction which facilitates clinical thought processes. However, problem-oriented documentation is challenging to put into practice using commercially available electronic health record (EHR) systems.</p><p><strong>Objective: </strong>Our goal was to create, iterate, and distribute a problem-oriented documentation toolkit within a commercial EHR that maximally supported clinicians' thinking, was intuitive to use, and produced clear documentation.</p><p><strong>Materials and methods: </strong>We used an iterative design process that stressed visual simplicity, data integration, a predictable interface, data reuse, and clinician efficiency. Creation of the problem-oriented documentation toolkit required the use of EHR-provided tools and custom programming.</p><p><strong>Results: </strong>We developed a problem-oriented documentation interface with a 3-column view showing (1) a list of visit diagnoses, (2) the current overview and assessment and plan for a selected diagnosis, and (3) a list of medications, labs, data, and orders relevant to that diagnosis. We also created a series of macros to bring information collected through the interface into clinicians' notes. This toolkit was put into a live environment in February 2019. Over the first 9 months, the custom problem-oriented documentation toolkit was used in a total of 8385 discrete visits by 28 clinicians in 13 ambulatory departments. After 9 months, the go-live education and EHR optimization teams in our health system began promoting the toolkit to new and existing users of our EHR resulting in a significantly increased uptake by outpatient clinicians. In April 2022 alone, the toolkit was used in more than 92 000 ambulatory visits by 894 users in 271 departments across our health system.</p><p><strong>Conclusions: </strong>As a health-system client of a commercial EHR, we developed and deployed a revised problem-oriented documentation toolkit that is used by clinicians more than 92 000 times a month. Key success elements include an emphasis on usability and an effective training effort.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"6 1","pages":"ooad005"},"PeriodicalIF":2.1,"publicationDate":"2023-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9897179/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"10672223","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A platform for phenotyping disease progression and associated longitudinal risk factors in large-scale EHRs, with application to incident diabetes complications in the UK Biobank.","authors":"Do Hyun Kim,&nbsp;Aubrey Jensen,&nbsp;Kelly Jones,&nbsp;Sridharan Raghavan,&nbsp;Lawrence S Phillips,&nbsp;Adriana Hung,&nbsp;Yan V Sun,&nbsp;Gang Li,&nbsp;Peter Reaven,&nbsp;Hua Zhou,&nbsp;Jin J Zhou","doi":"10.1093/jamiaopen/ooad006","DOIUrl":"https://doi.org/10.1093/jamiaopen/ooad006","url":null,"abstract":"<p><strong>Objective: </strong>Modern healthcare data reflect massive multi-level and multi-scale information collected over many years. The majority of the existing phenotyping algorithms use case-control definitions of disease. This paper aims to study the time to disease onset and progression and identify the time-varying risk factors that drive them.</p><p><strong>Materials and methods: </strong>We developed an algorithmic approach to phenotyping the incidence of diseases by consolidating data sources from the UK Biobank (UKB), including primary care electronic health records (EHRs). We focused on defining events, event dates, and their censoring time, including relevant terms and existing phenotypes, excluding generic, rare, or semantically distant terms, forward-mapping terminology terms, and expert review. We applied our approach to phenotyping diabetes complications, including a composite cardiovascular disease (CVD) outcome, diabetic kidney disease (DKD), and diabetic retinopathy (DR), in the UKB study.</p><p><strong>Results: </strong>We identified 49 049 participants with diabetes. Among them, 1023 had type 1 diabetes (T1D), and 40 193 had type 2 diabetes (T2D). A total of 23 833 diabetes subjects had linked primary care records. There were 3237, 3113, and 4922 patients with CVD, DKD, and DR events, respectively. The risk prediction performance for each outcome was assessed, and our results are consistent with the prediction area under the ROC (receiver operating characteristic) curve (AUC) of standard risk prediction models using cohort studies.</p><p><strong>Discussion and conclusion: </strong>Our publicly available pipeline and platform enable streamlined curation of incidence events, identification of time-varying risk factors underlying disease progression, and the definition of a relevant cohort for time-to-event analyses. These important steps need to be considered simultaneously to study disease progression.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"6 1","pages":"ooad006"},"PeriodicalIF":2.1,"publicationDate":"2023-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9912368/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"9974451","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Using artificial intelligence in medical school admissions screening to decrease inter- and intra-observer variability.","authors":"Graham Keir,&nbsp;Willie Hu,&nbsp;Christopher G Filippi,&nbsp;Lisa Ellenbogen,&nbsp;Rona Woldenberg","doi":"10.1093/jamiaopen/ooad011","DOIUrl":"https://doi.org/10.1093/jamiaopen/ooad011","url":null,"abstract":"<p><strong>Objectives: </strong>Inter- and intra-observer variability is a concern for medical school admissions. Artificial intelligence (AI) may present an opportunity to apply a fair standard to all applicants systematically and yet maintain sensitivity to nuances that have been a part of traditional screening methods.</p><p><strong>Material and methods: </strong>Data from 5 years of medical school applications were retrospectively accrued and analyzed. The applicants (<i>m</i> = 22 258 applicants) were split 60%-20%-20% into a training set (<i>m</i> = 13 354), validation set (<i>m</i> = 4452), and test set (<i>m</i> = 4452). An AI model was trained and evaluated with the ground truth being whether a given applicant was invited for an interview. In addition, a \"real-world\" evaluation was conducted simultaneously within an admissions cycle to observe how it would perform if utilized.</p><p><strong>Results: </strong>The algorithm had an accuracy of 95% on the training set, 88% on the validation set, and 88% on the test set. The area under the curve of the test set was 0.93. The SHapely Additive exPlanations (SHAP) values demonstrated that the model utilizes features in a concordant manner with current admissions rubrics. By using a combined human and AI evaluation process, the accuracy of the process was demonstrated to be 96% on the \"real-world\" evaluation with a negative predictive value of 0.97.</p><p><strong>Discussion and conclusion: </strong>These results demonstrate the feasibility of an AI approach applied to medical school admissions screening decision-making. Model explainability and supplemental analyses help ensure that the model makes decisions as intended.</p>","PeriodicalId":36278,"journal":{"name":"JAMIA Open","volume":"6 1","pages":"ooad011"},"PeriodicalIF":2.1,"publicationDate":"2023-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9936956/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"10768567","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}