Health AI poses distinct harms and potential benefits for disabled people

Abstract

Relative to other groups for whom the risk of bias in artificial intelligence (AI) has been identified, little attention has been given to the potential harms, and distinct benefits, that AI models could bring about for disabled people. Predictions made by AI models for disabled patients may lead to discrimination in at least three ways: disabled patients are underrepresented in datasets; they have historically been discriminated against in their medical care; and data used to predict physiological frailty may overestimate the degree of frailty for disabled patients.

The disability community is smaller than the general population, and most healthcare data on which AI models are being trained come from non-disabled patients. AI models trained on data primarily from non-disabled patients may accurately predict a given outcome for these patients but fail when applied to disabled patients. In addition, health data pertinent to disabled patients may be missing or incomplete. The potential for bias and discrimination attributable to underrepresentation and misrepresentation in datasets is well described¹.