AR Food Changer using Deep Learning And Cross-Modal Effects

Abstract

We propose an AR application that enables us to change the appearance of food without AR markers by applying machine learning and image processing. Modifying the appearance of real food is a difficult task because the shape of the food is atypical and deforms while eating. Therefore, we developed a real-time object region extraction method that combines two approaches in a complementary manner to extract food regions with high accuracy and stability. These approaches are based on color and edge information processing with a deep learning module trained with a small amount of data. Besides, we implemented some novel methods to improve the accuracy and reliability of the system. Then, we experimented and the results show that the taste and oral texture were affected by visual textures. Our application can change not only the appearance in real-time but also the taste and texture of actual real food. Therefore, in conclusion, our application can be virtually termed as an "AR food changer".