Restaurant recommendations under multimodal online reviews: A novel method based on image captioning and text analysis with multi-criteria decision-making

Abstract

Restaurant selection has become a complex decision-making process for consumers, driven by an overwhelming volume of online reviews. While text and numerical reviews provide valuable insights, the increasing use of visual content, further enriches consumer evaluations. However, existing research lacks effective methods for integrating multimodal reviews to facilitate informed decision-making. To address this gap, this paper proposes a novel approach for restaurant selection based on multimodal online reviews, the contributions of which mainly focus on the following aspects: (i) employ image captioning techniques to convert image review into textual descriptions, bridging the gap between image and text, (ii) apply text analysis methods to extract relevant evaluation criteria from both text and image-generated descriptions, and (iii) integrate insights from both modalities by assessing the object and content consistency between image and text, ensuring the reliability of reviews. The method is applied to Yelp, using a dataset of 31,412 reviews from 10 restaurants. Eight evaluation criteria are extracted from both text and image reviews. The results show that compared with single-modal and dual-modal review-based recommendation methods, the proposed multimodal approach uncovers more comprehensive evaluation criteria and generates more realistic ranking results. Additionally, the proposed information fusion method outperforms traditional fusion methods in effectively integrating multimodal information.