Recipe Popularity Prediction with Deep Visual-Semantic Fusion

Predicting the popularity of user-created recipes has great potential to be adopted in several applications on recipe-sharing websites. To ensure timely prediction when a recipe is uploaded, a prediction model needs to be trained based on the recipe's content features (i.e., its visual and semantic features). This paper presents a novel approach to predicting recipe popularity using deep visual-semantic fusion. We first pre-train a deep model that predicts the popularity of recipes based on each single modality. We insert additional layers to the two models and concatenate their activations. Finally, we train a network comprising fully connected (FC) layers on the fused features to learn more powerful features, which are used for training a regressor. Based on experiments conducted on more than 150K recipes collected from the Cookpad website, we present a comprehensive comparison with several baselines to verify the effectiveness of our method. The best practice for the proposed method is also described.