Quality Review of Mass Adjudication: A Randomized Natural Experiment at the Board of Veterans Appeals, 2003–16

We study a unique natural experiment, during which 5-10% of draft opinions by judges of the Board of Veterans Appeals (BVA) were randomly selected for “quality review” by a team of full-time staff attorneys for nearly 15 years. This performance program had the express goals of measuring accuracy and reducing reversal rates on appeal. In cases of legal error, the quality review team wrote memoranda to judges to permit correction before opinions were issued. We use rich internal administrative data on nearly 600,000 cases from 2002-2016 to provide the first rigorous study of this review process. With precise estimates, we show that the program had no appreciable effect on reducing appeals or reversals. Based on internal records, we demonstrate that this inefficacy is likely by design, as meeting the performance measure of “accuracy” was at cross-purposes with error correction. These findings inform longstanding questions of law, organization, and bureaucracy, including performance management, standards of review, and the institutional design of mass adjudication.