Towards a transfer learning approach to food recommendations through food images

Abstract

User generated text/multimedia content are increasingly shared in online businesses systems and their effective use in user modelling and recommendation strategies is consequently growing too. In restaurant businesses the food menu along with images are a common practice and users also shared the food images they ordered and feel good about. Yelp data set challenge in their round 9 and onward introduced such rich images data for their competition. In this paper, we motivated from this rich images data of food for semantically incorporating image-specific features to the star-rating and recommendation process. We first applied a transfer learning approach with pre-trained CNNs (Convolution Neural Networks) which were used to label the Yelps food images of the restaurants using the Food101 data-set. We defined star-rating for restaurants by capturing a correlation between restaurant images and users shared images. Our proposed strategy works on discovering hidden aspects of food images and labels to be used in recommendation strategy. We performed an extensive set of experiments by creating a baseline using standard rating provided in the Yelp data-set. The proposed approach produced better Root Mean Square Error (RMSE), which is a clear indication of high-quality recommendation strategy