Windows Attention Based Pyramid Network for Food Segmentation

Abstract

Recently, food segmentation has obtained growing attention in the field of computer vision for its great potential in human health. Most of existing methods utilize deep visual features extracting from Convolutional Neural Networks (CNNs) for food segmentation. However, these works ignore characteristics of food images and are thus difficult to achieve optimal segmentation performance. Compared with general image segmentation, food images usually do not exhibit unique spatial layout and common semantic patterns. In this paper, we address the food image segmentation task by capturing richer contextual and boundary information. The previous works capture image representation by multi-scale feature fusion, we propose a Windows Attention based Pyramid Network (WAPNet) to adaptively combine local features with global dependencies. Specifically, WAPNet combines Feature Pyramid Network (FPN) with Window Attention to weight multi-scale features, and then extract richer marginal information. In addition, we utilize a multimodality pre-training approach Recipe Learning Module (ReLeM) that explicitly provides segmentation model with rich semantic food knowledge. And by introducing Locality and Windows design, calculating self-attention according to Windows, We demonstrate promising performance on a new proposed food image benchmark for semantic segmentation.