Auditing Algorithms: Understanding Algorithmic Systems from the Outside In

Abstract

Algorithms are ubiquitous and critical sources of information online, increasingly acting as gatekeepers for users accessing or sharing information about virtually any topic, including their personal lives and those of friends and family, news and politics, entertainment, and even information about health and well-being. As a result, algorithmically-curated content is drawing increased attention and scrutiny from users, the media, and lawmakers alike. However, studying such content poses considerable challenges, as it is both dynamic and ephemeral: these algorithms are constantly changing, and frequently changing silently, with no record of the content to which users have been exposed over time. One strategy that has proven effective is the algorithm audit: a method of repeatedly querying an algorithm and observing its output in order to draw conclusions about the algorithm’s opaque Danaë Metaxa, Joon Sung Park, Ronald E. Robertson, Karrie Karahalios, Christo Wilson, Jeff Hancock and Christian Sandvig (2021), “Auditing Algorithms”, Foundations and Trends® in Human-Computer Interaction: Vol. 14, No. 4, pp 272–344. DOI: 10.1561/1100000083. ©2021 D. Metaxa et al.