Mixed Methods Machine Learning

Machine learning is ubiquitous: many of our everyday interactions, both online and offline, are backed by machine learning. Typically, machine learned systems start as an idea from the business or engineering team for a service or an app that helps the customer achieve a goal. The app is built iteratively, starting with the minimum lovable version, and undergoes several rounds of improvements to become more sophisticated. Success is measured with an online A/B test on live traffic, on the assumption that if customers engage with the app, it is serving their needs. We propose a different approach to developing such systems, that employs mixed-methods research to understand what to build, and how to make it satisfying and helpful for the customer. The Mixed Methods Machine Learning (MXML) paradigm, starts with a user study, to understand how people behave in an everyday setting (such as shopping for groceries in a grocery store), and to identify points of friction that can be automated, or experiences that can be made more enjoyable. The study observations are mapped to interactions recorded in the system's behavioral log data, which is the basis for the machine learned system. Mapping the study observations to the log data is a key step in directing the machine learning to solve a customer problem. The MXML system is evaluated with a follow-on user study, in addition to the traditional online A/B test, to assess whether the system is satisfying, helpful and delightful. In this talk we present the MXML paradigm, with real-world examples.