Multi-task visual food recognition by integrating an ontology supported with LLM

Abstract

Food image analysis is a crucial task with far-reaching implications across various domains, including culinary arts, nutrition, and food technology. This paper presents a novel approach to multi-task visual food analysis, using large language models to obtain recipes and support the creation of a comprehensive food ontology. The approach integrates the food ontology into an end-to-end model, with prior knowledge on the relationships of food concepts at different semantic levels, within a multi-task deep learning visual food analysis approach, to generate better and more consistent class predictions. Evaluated on two benchmark datasets, MAFood-121 and VireoFood-172, this method demonstrates its effectiveness in single-label food recognition and multi-label food group classification. The ontology enhances accuracy, consistency, and generalization by effectively transferring knowledge to the learning model. This study underscores the potential of ontology-based methods to address food image classification complexities, with implications for broad applications, including automated recipe generation and nutritional assessment.