Context awareness in graph-based image semantic segmentation via visual word distributions

Abstract

This paper addresses the problem of image semantic segmentation (or semantic labelling), that is the association of one of a predefined set of semantic categories (e.g. cow, car, face) to each image pixel. We adopt a patch-based approach, in which super-pixel elements are obtained via oversegmentation of the original image. We then train a Conditional Random Field on heterogeneous descriptors extracted at different scales and locations. This discriminative graphical model can effectively account for the statistical dependence of neighbouring patches. For the more challenging task of considering long-range patch dependency and contextualisation, we propose the use of a descriptor based on histograms of visual words extracted in the vicinity of each patch at different scales. Experiments validate our approach by showing improvements with respect to both a base model not using distributed features and the state of the art works in the area.