Multimodal Dish Pairing: Predicting Side Dishes to Serve with a Main Dish

Abstract

Planning a food menu is an essential task in our daily lives. We need to plan a menu by considering various perspectives. To reduce the burden when planning a menu, this study first tackles a novel problem of multimodal dish pairing (MDP), i.e., retrieving suitable side dishes given a query main dish. The key challenge of MDP is to learn human subjectivity, i.e., one-to-many relationships of the main and side dishes. However, in general, web resources only include one-to-one manually created pairs of main and side dishes. To tackle this problem, this study assumes that if side dishes are similar to a manually created side dish, they are also acceptable for the query main dish. We then imitate a one-to-many relationship by computing the similarity of side dishes as side dish scores and assigning them to unknown main and side dish pairs. Based on this score, we train a neural network to learn the suitability of the side dishes through learning-to-rank techniques by fully leveraging the multimodal representations of the dishes. During the experiments, we created a dataset by crawling recipes from an online menu site and evaluated the proposed method based on five criteria: retrieval evaluation, overlapping ingredients, overlapping cooking methods, consistency of the dish styles, and human evaluations. Our experiment results show that the proposed method is superior to the baseline in terms of these five criteria. The results of the qualitative analysis further demonstrates that the proposed method can retrieve side dishes suitable for the main dish.