Estimating expert prior knowledge from optimization trajectories

Abstract

A recurring task in research is iterative optimization of a process that can be evaluated only by conducting an experiment. Powerful algorithms for assisting this process exist, but they largely ignore the valuable knowledge of expert scientists. We consider a problem within this general scope, not aiming to automate the optimization but instead studying how to infer tacit expert knowledge. This complements the current literature focusing on how such information is used in the optimization process, paying little attention on how the information is obtained. We consider a new formulation where the expertise is inferred by passively observing a human solving an optimization problem, without requiring explicit elicitation techniques. Our solution leverages concepts from Bayesian optimization (BO) commonly used for automating the optimization, but now these tools are used as a theoretical model for the user behavior instead. We assume the expert solves the task approximately in the same manner as a BO algorithm would and solve what kind of prior knowledge about the target function is consistent with the sequence of choices they made. We introduce the problem and a concrete solution, and show that the recovered priors match the true priors in controlled simulated studies. We also empirically evaluate the robustness of the method against violations of the modeling assumptions and demonstrate it on real user data.