Samuel W Fenske, Alec Peltekian, Mengija Kang, Nikolay S Markov, Mengou Zhu, Kevin Grudzinski, Melissa J Bak, Anna Pawlowski, Vishu Gupta, Yuwei Mao, Stanislav Bratchikov, Thomas Stoeger, Luke V Rasmussen, Alok N Choudhary, Alexander V Misharin, Benjamin D Singer, GR Scott Budinger, Richard D Wunderink, Ankit Agrawal, Catherine A Gao, NU Script Study Investigators
{"title":"Developing and validating a machine learning model to predict successful next-day extubation in the ICU","authors":"Samuel W Fenske, Alec Peltekian, Mengija Kang, Nikolay S Markov, Mengou Zhu, Kevin Grudzinski, Melissa J Bak, Anna Pawlowski, Vishu Gupta, Yuwei Mao, Stanislav Bratchikov, Thomas Stoeger, Luke V Rasmussen, Alok N Choudhary, Alexander V Misharin, Benjamin D Singer, GR Scott Budinger, Richard D Wunderink, Ankit Agrawal, Catherine A Gao, NU Script Study Investigators","doi":"10.1101/2024.06.28.24309547","DOIUrl":null,"url":null,"abstract":"Background: Criteria to identify patients who are ready to be liberated from mechanical ventilation are imprecise, often\nresulting in prolonged mechanical ventilation or reintubation, both of which are associated with adverse outcomes. Daily\nprotocol-driven assessment of the need for mechanical ventilation leads to earlier extubation but requires dedicated\npersonnel. We sought to determine whether machine learning applied to the electronic health record could predict\nsuccessful extubation.\nMethods: We examined 37 clinical features from patients from a single-center prospective cohort study of patients in our\nquaternary care medical ICU who required mechanical ventilation and underwent a bronchoalveolar lavage for known or\nsuspected pneumonia. We also tested our models on an external test set from a community hospital ICU in our health care\nsystem. We curated electronic health record data aggregated from midnight to 8AM and labeled extubation status. We\ndeployed three data encoding/imputation strategies and built XGBoost, LightGBM, logistic regression, LSTM, and RNN\nmodels to predict successful next-day extubation. We evaluated each model's performance using Area Under the Receiver\nOperating Characteristic (AUROC), Area Under the Precision Recall Curve (AUPRC), Sensitivity (Recall), Specificity, PPV\n(Precision), Accuracy, and F1-Score.\nResults: Our internal cohort included 696 patients and 9,828 ICU days, and our external cohort had 333 patients and 2,835\nICU days. The best model (LSTM) predicted successful extubation on a given ICU day with an AUROC 0.87 (95% CI 0.834-\n0.902) and the internal test set and 0.87 (95% CI 0.848-0.885) on the external test set. A Logistic Regression model\nperformed similarly (AUROC 0.86 internal test, 0.83 external test). Across multiple model types, measures previously\ndemonstrated to be important in determining readiness for extubation were found to be most informative, including plateau\npressure and Richmond Agitation Sedation Scale (RASS) score. Our model often predicted patients to be stable for\nextubation in the days preceding their actual extubation, with 63.8% of predicted extubations occurring within three days of\ntrue extubation. We also tested the best model on cases of failed extubations (requiring reintubation within two days) not\nseen by the model during training. Our best model would have identified 35.4% (17/48) of these cases in the internal test\nset and 48.1% (13/27) cases in the external test set as unlikely to be successfully extubated.\nConclusions: Machine learning models can accurately predict the likelihood of extubation on a given ICU day from data\navailable in the electronic health record. Predictions from these models are driven by clinical features that have been\nassociated with successful extubation in clinical trials.","PeriodicalId":501249,"journal":{"name":"medRxiv - Intensive Care and Critical Care Medicine","volume":"13 1","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2024-06-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"medRxiv - Intensive Care and Critical Care Medicine","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1101/2024.06.28.24309547","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0
Abstract
Background: Criteria to identify patients who are ready to be liberated from mechanical ventilation are imprecise, often
resulting in prolonged mechanical ventilation or reintubation, both of which are associated with adverse outcomes. Daily
protocol-driven assessment of the need for mechanical ventilation leads to earlier extubation but requires dedicated
personnel. We sought to determine whether machine learning applied to the electronic health record could predict
successful extubation.
Methods: We examined 37 clinical features from patients from a single-center prospective cohort study of patients in our
quaternary care medical ICU who required mechanical ventilation and underwent a bronchoalveolar lavage for known or
suspected pneumonia. We also tested our models on an external test set from a community hospital ICU in our health care
system. We curated electronic health record data aggregated from midnight to 8AM and labeled extubation status. We
deployed three data encoding/imputation strategies and built XGBoost, LightGBM, logistic regression, LSTM, and RNN
models to predict successful next-day extubation. We evaluated each model's performance using Area Under the Receiver
Operating Characteristic (AUROC), Area Under the Precision Recall Curve (AUPRC), Sensitivity (Recall), Specificity, PPV
(Precision), Accuracy, and F1-Score.
Results: Our internal cohort included 696 patients and 9,828 ICU days, and our external cohort had 333 patients and 2,835
ICU days. The best model (LSTM) predicted successful extubation on a given ICU day with an AUROC 0.87 (95% CI 0.834-
0.902) and the internal test set and 0.87 (95% CI 0.848-0.885) on the external test set. A Logistic Regression model
performed similarly (AUROC 0.86 internal test, 0.83 external test). Across multiple model types, measures previously
demonstrated to be important in determining readiness for extubation were found to be most informative, including plateau
pressure and Richmond Agitation Sedation Scale (RASS) score. Our model often predicted patients to be stable for
extubation in the days preceding their actual extubation, with 63.8% of predicted extubations occurring within three days of
true extubation. We also tested the best model on cases of failed extubations (requiring reintubation within two days) not
seen by the model during training. Our best model would have identified 35.4% (17/48) of these cases in the internal test
set and 48.1% (13/27) cases in the external test set as unlikely to be successfully extubated.
Conclusions: Machine learning models can accurately predict the likelihood of extubation on a given ICU day from data
available in the electronic health record. Predictions from these models are driven by clinical features that have been
associated with successful extubation in clinical trials.