Learning about AI ethics from cases: a scoping review of AI incident repositories and cases

Abstract

Cases provide a practical resource for learning regarding the uses and challenges of AI applications. Cases give insight into how principles and values are implicated in real contexts, the trade-offs and different perspectives held regarding these contexts, and the—sometimes hidden—relationships between cases, relationships that may support analogical reasoning across contexts. We aim to (1) provide an approach for structuring ethics cases and (2) investigate existing case repository structures. We motivate a scoping review through a conceptual analysis of ethics case desirable features. The review sought to retrieve repositories, (sometimes known as observatories, catalogues, galleries, or incident databases), and their cases, for analysis of their expression of ethics concepts. We identify n = 14 repositories, extracting the case schema used in each, to identify how this metadata can express ethical concepts. We find that most repositories focus on harm-indicators, with some indicating positive impacts, but with little explicit reference to ethical concepts; a subset (n = 4) includes no structural elements addressing ethical concepts or impacts. We extract a subset of cases from the total cases (n = 2000) across repositories addressing education (n = 100). These are grouped by topic, with a structured content analysis provided of ethical implications from one sub-theme, offering qualitative insights into the ethical coverage. Our conceptual analysis and empirical review exemplify a model for ethics cases (shorthanded as Ethics-case-CPR), while highlighting gaps both in existing case repositories and specific examples of cases.