Machine Advice with a Warning about Machine Limitations: Experimentally Testing the Solution Mandated by the Wisconsin Supreme Court

Abstract

The Wisconsin Supreme Court allows machine advice in the courtroom only if accompanied by a series of warnings. We test 878 US lay participants with jury experience on fifty past cases where we know ground truth. The warnings affect their estimates of the likelihood of recidivism and their confidence, but not their decision whether to grant bail. Participants do not get better at identifying defendants who recidivated during the next two years. Results are essentially the same if participants are warned in easily accessible language, and if they are additionally informed about the low accuracy of machine predictions. The decision to grant bail is also unaffected by the warnings mandated by the Supreme Court if participants do not first decide without knowing the machine prediction. Oversampling cases where defendants committed violent crime does not change results either, whether coupled with machine predictions for general or for violent crime. Giving participants feedback and incentivizing them for finding ground truth has a small, weakly significant effect. The effect becomes significant at conventional levels when additionally using strong graphical warnings. Then participants are less likely to follow the advice. But the effect is counterproductive: they follow the advice less if it actually is closer to ground truth.