Predicting and preferring

ABSTRACTThe use of machine learning, or ‘artificial intelligence’ (AI) in medicine is widespread and growing. In this paper, I focus on a specific proposed clinical application of AI: using models to predict incapacitated patients’ treatment preferences. Drawing on results from machine learning, I argue this proposal faces a special moral problem. Machine learning researchers owe us assurance on this front before experimental research can proceed. In my conclusion I connect this concern to broader issues in AI safety.KEYWORDS: PPPmedical ethicsAIpatient preference predictorspreference shaping Disclosure statementNo potential conflict of interest was reported by the author(s).Notes1 In what follows, I mostly refer to these systems as ML systems, rather than as ‘AI’, in order to avoid unfortunate and controversial implications about machine ‘intelligence’.2 For an overview, see (Emanuel et al. Citation1991; Buchanan and Brock Citation2019).3 For discussion, see (Salmond and David Citation2005; Shalowitz, Garrett-Mayer, and Wendler Citation2006; Jezewski et al. Citation2007).4 See (Rid and Wendler Citation2014a) for discussion; for relevant machine learning research, see (O. Evans et al. Citation2018).5 For a selection of moral criticism, see (N. Sharadin Citation2019; N. P. Sharadin Citation2018; Ditto and Clark Citation2014; Kim Citation2014; John Citation2014; Dresser Citation2014; Tretter and Samhammer Citation2023; Mainz Citation2022). For a recent reply to autonomy-based criticism, see (Jardas et al. Citation2022).6 Compare (N. P. Sharadin Citation2018).7 For a technical overview, see (Gneiting and Raftery Citation2007).8 Well, three. We could change our scoring rule, or our performance metric. I ignore this possibility in what follows.9 I follow the literature in saying that a learner is incentivized to do something just in case doing that thing increases performance (or reward). See (Krueger, Maharaj, and Leike Citation2020, 2).10 If this sounds familiar from the Forever War between consequentialists and Kantians, that’s not an accident.11 See (Good Citation2021).12 Philosophers call a related phenomenon self-fulfilling beliefs (Silva Citationforthcoming; Antill Citation2019).13 Following (Perdomo et al. Citation2020). Begrudgingly because it can make it sound as if the model itself is doing something. It isn’t: we are doing something with the model.14 Compare (Franklin et al. Citation2022).15 This follows from broader ideas about the importance of informed consent. For an overview, see (Faden and Beauchamp Citation1986); for critical discussion, see (Manson and O’Neill Citation2007).16 This is not controversial. See (Li and Chapman Citation2020) for discussion.17 For a recent philosophical discussion, see Parmer (Citation2023). The debate over the ethics of nudging is ongoing. For the classic source on ‘nudges’ see Thaler and Sunstein (Citation2008).18 For technical discussion of the broad phenomenon, see (Krueger, Maharaj, and Leike Citation2020; C. Evans and Kasirzadeh Citation2022; Farquhar, Carey, and Everitt Citation2022; Everitt et al. Citation2021).19 See the discussion in (Perdomo et al. Citation2020).20 Thanks to an anonymous referee for encouraging clarity on this point.21 The only research that I’m aware of that approaches the question of performative prediction in the context of medical AI is a review article (Chen et al. Citation2021); there, the authors simply note the possibility of distributional shift (aka performative prediction).22 This is also the conclusion of other AI safety researchers. Compare (Hendrycks et al. Citation2022; C. Evans and Kasirzadeh Citation2022; Ashton and Franklin Citation2022). This is not to say that there are no proposals about how to ensure that models have other interesting properties related to performative prediction, e.g., can achieve various strategic equilibria; for relevant discussion see (Mendler-Dünner et al. Citation2020; Brown, Hod, and Kalemaj Citation2022; Miller, Perdomo, and Zrnic Citation2021). Very recent work aims to identify and penalize induced preference shifts in recommender systems (e.g. Carroll et al. Citation2022); that work is clearly relevant to the present problem, though it doesn’t yet represent a solution.23 This is part of why no one agrees on a definition of the Alignment Problem.24 Thanks to an anonymous referee for this way of putting the contrast between the alignment problem and the problem I identify in the paper.25 Thanks to Simon Goldstein, Dan Hendrycks, Jacqueline Harding, Cameron Kirk-Giannini, David Krueger, Nick Laskowski, Robert Long, Elliot Thornley, and members of the Cottage Group for helpful discussions about these and related issues.