A Data-Driven Approach to Beating SAA Out of Sample

Abstract

A Little Pessimism Goes a Long Way Data-driven optimization is concerned with finding a decision, using data and perhaps a model, that performs well when it is applied on a new unseen data point. Data-driven optimization is challenging because data are limited or the model is wrong or the environment in which the decision is being applied is different from the one in which the training data were collected. Distributionally robust optimization (DRO), a worst case optimization method for finding decisions that are insensitive to model error, can sometimes but not always deliver a decision that has a larger out-of-sample expected reward than the sample average approximation (SAA). “A Data Driven Approach to Beating SAA out of Sample” by Jun-ya Gotoh, Michael Kim, and Andrew Lim shows that if worst case (DRO) solutions fail at this task, then the solution of a best case distributionally optimistic optimization problem will do the job. As good as this sounds, there is a catch: whereas an optimistic decision might beat SAA, the improvement is very modest and comes at the cost of being much more sensitive to model misspecification than both the SAA and the DRO decisions. Moreover, it is easy to make a mistake: it can be difficult to determine with a modestly sized data set whether the best or worst case solution will have the higher expected reward than SAA. In summary, data driven optimization is a trade-off between maximizing the expected reward and controlling the sensitivity of this expectation to model misspecification. When both are considered, a little bit of pessimism goes a long way.