3D Mesh Reconstruction of Foods from a Single Image

Abstract

Dietary calorie management has been an important topic in recent years, and various methods and applications on image-based food calorie estimation have been published in the multimedia community. Most of the existing methods of estimating food calorie amounts use 2D-based image recognition. On the other hand, in this extended abstract, we would like to introduce our work on 3D food volume estimation employing a recent DNN-based 3D mesh reconstruction technique. We performed 3D mesh reconstruction of a dish~(food and plate) and a plate (without foods) from a single image. We succeeded in restoring the 3D shape with high accuracy while maintaining the consistency between a plate part of an estimated 3D dish and an estimated 3D plate. To achieve this, the following contributions were made in our recent work. (1) Proposal of "Hungry Networks,'' a new network that generates two kinds of 3D volumes from a single image. (2) Introduction of plate consistency loss that matches the shapes of the plate parts of the two reconstructed models. (3) Creating a new dataset of 3D food models that are 3D scanned of actual foods and plates. We also conducted an experiment to infer the volume of only the food region from the difference of the two reconstructed volumes. As a result, it was shown that the introduced new loss function not only matches the 3D shape of the plate, but also contributes to obtaining the volume with higher accuracy. Although there are some existing studies that consider 3D shapes of foods, this is the first study to generate a 3D mesh volume from a single dish image. In addition, we have implemented a web-based 3D dish reconstruction system, "Pop'n Food'', which enables reconstruction of 3D shapes from a single dish image in a real-time way. The demo video of the system is available at https://youtu.be/YyIu8bL65EE.