Balancing Expert Opinion and Historical Data: The Case of Baseball Umpires

Abstract

Many decisions benefit from situations where there exist both ample expert opinion and historical data. In cost modeling these may include the costs of software development, the learning curve rates for specific manufacturing tasks, and the unit rate costs of operating certain products. When making forecasts we are often faced with the decision to base our estimates on either expert opinion or historical data. When these two perspectives converge, we have high confidence in the estimate. The more interesting case is when they contradict. This is where the estimator needs to dig deeper in order to determine the sources of inconsistencies. Cost modelers are not the only ones who struggle with deciding whether to trust experts or data. Data scientists are increasingly dealing with this duality especially in the context of professional sports where expert opinion is associated with the traditional viewpoint and data-driven decision making is associated with a more modern approach. In the United States, professional sports teams are increasingly using analytics to optimize their athletes’ performance as well as their business operations (Pelton, 2015). But the culture of professional sports still depends heavily on experience and gut feel. The case of baseball umpires provides a good example of expert opinion being preferred over historical data. In professional baseball, the umpire’s job is to determine whether the ball passed the strike zone1 or not. If the batter does not swing it is left to the umpire’s expert judgement to identify whether the pitch was a ball or a strike. The strike zone is defined in the official rules of baseball and are not subject to interpretation, however, the implementation of measuring said strike zone is entirely left to human judgement. Even more challenging is that the decision must be made in a matter of seconds under extreme pressure. Chen, Moskowitz, and Shue (2016) analyzed baseball umpire data using the PITCHf/x system that tracks the actual location of each pitch using multiple cameras. By comparing the umpire’s decision to the actual placement of the ball relative to the strike zone they determined that, during the 2008 to 2016 seasons which included 127 different umpires calling over 3.5 million pitches, umpires were correct only part of the time as shown in Table 1. If baseball umpires are getting one out of every eight ball/strike calls wrong, this adds up to more than 30,000 mistakes a year. In most industries, and even other professional sports leagues, this would be unacceptable but baseball traditionalists are hesitant to adopt new technologies that remove the human element from the game.