Consideration of Sociodemographics in Machine Learning-Driven Sepsis Risk Prediction.

Abstract

Objectives: Use of machine learning (ML) and artificial intelligence (AI) in prediction of sepsis and related outcomes is growing. Guidelines call for explicit reporting of study data demographics and stratified performance analyses to assess potential sociodemographic bias. We assessed reporting of sociodemographic data and other considerations, such as use of stratified analyses or use of so-call "fairness metrics", among AI and ML models in sepsis.

Data sources: PubMed identified systematic and narrative reviews from which studies were extracted using PubMed and Google Scholar.

Study selection: Studies were extracted from selected review articles published between January 1, 2023, and June 30, 2024, and related to sepsis, risk prediction, and ML; we extracted studies predicting sepsis, sepsis-related outcomes, or sepsis treatment in adult populations.

Data extraction: Data were extracted by two reviewers using predefined forms, and included study type, outcome of interest, setting, dataset used, reporting of sample sociodemographics, inclusion of sociodemographics as predictors, stratification by sociodemographics or assessment of fairness metrics, and reporting a lack of sociodemographic considerations as a limitation.

Data synthesis: Thirteen of 96 review studies (14%) met inclusion criteria: six systematic reviews and seven narrative reviews. One hundred twenty of 170 studies (71%) extracted from these review articles were included in our review. Ninety-nine of 120 studies (83%) reported a measure of geography or where data was collected. Eighty (67%) reported sex/gender, 24 (20%) reported race/ethnicity, and 4 (3%) reported other sociodemographics. Only three stratified performance results (2%) by sociodemographics; none reported formal fairness metrics. Beyond a lack of geographic heterogeneity (39/120, 33%), few studies reported a lack of sociodemographic consideration as a limitation.

Conclusions: The inclusion of sociodemographic data and stratified assessment of performance-essential steps in developing equitable risk prediction tools-are possible but have yet to be consistently adopted.