Aggregation, trade-offs, and uncertainties in AI wellbeing

Abstract

This paper examines how, if artificial agents are capable of wellbeing, their wellbeing should be compared and aggregated alongside human wellbeing. Building on arguments from Goldstein and Kirk-Giannini, who suggest that some AI systems may possess wellbeing, I explore the moral implications of this possibility. Rather than reinventing debates in population ethics, this paper adapts and extends them to the context of AI wellbeing. I analyse three major approaches to wellbeing aggregation: symmetric methods, which treat human and AI wellbeing as equally significant; uncertainty-responsive methods, which discount AI wellbeing due to ontological, temporal, or identity uncertainty; and constraint-based views, which impose categorical constraints on trading off human wellbeing for AI gains. These approaches are tested against thought experiments involving classic problems, such as the repugnant conclusion, infinitarian paralysis, and fanaticism. While utilitarian approaches risk endorsing troubling consequences when AI wellbeing scales indefinitely, constraint-based views may underweight the wellbeing of AI. A distinctive finding is that  our intuitions shift depending on whether a human or an AI is a welfare subject. This reveals a potential asymmetry in our intuitive judgments, suggesting that species identity may itself be a morally salient feature that future theories of AI wellbeing should address. I conclude that uncertainty-responsive approaches, particularly those combining ontological, temporal, and identity-based discounting, offer a promising middle path that incorporates AI wellbeing into our moral calculus without letting it disproportionately outweigh human wellbeing in aggregation.