Construction of extra-large scale screening tools for risks of severe mental illnesses using real world healthcare data

Importance

The prevalence of severe mental illnesses (SMIs) in the United States is approximately 3 % of the whole population. The ability to conduct risk screening of SMIs at large scale could inform early prevention and treatment.

Objective

A scalable machine learning based tool was developed to conduct population-level risk screening for SMIs, including schizophrenia, schizoaffective disorders, psychosis, and bipolar disorders, using 1) healthcare insurance claims and 2) electronic health records (EHRs).

Design, setting and participants

Data from beneficiaries from a nationwide commercial healthcare insurer with 77.4 million members and data from patients from EHRs from eight academic hospitals based in the U.S. were used. First, the predictive models were constructed and tested using data in case-control cohorts from insurance claims or EHR data. Second, performance of the predictive models across data sources was analyzed. Third, as an illustrative application, the models were further trained to predict risks of SMIs among 18-year old young adults and individuals with substance associated conditions.

Main outcomes and measures

Machine learning-based predictive models for SMIs in the general population were built based on insurance claims and EHR.

Results

A total of 301,221 patients with SMIs and 2,439,890 control individuals were retrieved from the nationwide health insurance claim database in the U.S. A total of 59,319 patients with SMIs and 297,993 control individuals were retrieved from EHRs spanning eight different hospitals from a major integrated healthcare system in Massachusetts, U.S. The obtained predictive models for SMIs achieved AUCROC of 0.76, specificity of 79.1 % and sensitivity of 61.9 % on an independent test set of an all-age case-control cohort from insurance claim data, and AUCROC of 0.83, specificity of 85.1 % and sensitivity of 66.4 % using EHR data. The fine-tuned models for specific use case scenarios outperformed two rule based benchmark methods when predicting 12-month risk of SMIs among 18-year old young adults but had inferior performance to benchmark methods when predicting SMIs among individuals with substance associated conditions in claims data.

Conclusion

Performance of our SMI prediction models constructed using health insurance claims or EHR data suggest feasibility of using real world healthcare data for large scale screening of SMIs in the general population. In addition, our analysis showed cross data source generalizability of machine learning models trained on real world healthcare data. Models constructed from insurance claims appear to be transferable to EHR cohorts and vice versa.