Reply to the discussants of “Experimental Evaluation of Algorithm-Assisted Human Decision-Making: Application to Pretrial Public Safety Assessment.” Journal of the Royal Statistical Society, Series A, Forthcoming

Abstract

We thank the Research Section of the Royal Statistical Society for providing us with a valuable opportunity to present our paper and receive feedback from many scholars. Despite their diverse perspectives, we believe that all of our discussants are in agreement about the need to develop new statistical methodology to better analyze algorithm-assisted human decision making. While the use of algorithms has become ubiquitous in today’s society, we — humans — still make many consequential decisions with the help of algorithms. As Kumar and VanderWeele noted, such a hybrid decision-making system may allows one to combine human experiences and knowledge with algorithmic recommendations, possibly leading to improved decisions. Moreover, even if fully algorithmic decision-making is more optimal than a hybrid system, we may still prefer the latter because we want to hold humans, rather than algorithms, accountable for the consequences of decisions. The importance of algorithm-assisted human decision making motivated us to develop a set of methodological tools to evaluate and understand how algorithmic recommendations affect human decisions. As echoed by Cruz Cortéz and Gosh, our study sharply contrasts with much of the existing studies whose focus has been the accuracy and fairness of algorithms themselves. We hope other researchers follow up on this important research agenda. Below, we respond to specific comments raised by the discussants.