HFCER : Hybrid Fusion for Cultural Event Recognition in Images

Understanding high level semantic concepts in images requires information from various modalities of visual concepts. One such task is recognition of events based on still images, which requires simultaneous reasoning about high level semantic concepts like objects, people, scenes and their interactions. In this work, we explore different strategies to fuse object and scene information in images to aid the task of cultural event recognition. We start with early and late fusion strategies to combine object and scene level information to reason about event classes. To support our hypothesis that early fused models are able to extract complementary object and scene information, we propose the use of guided backpropagation method to visualize image activations. Inspection of image activations gives an essence of object-scene complementarity in case of early fusion which is not observed in the case of late fused models. As extensions to early and late fusion techniques, we propose HFCER, a hybrid fusion framework along with an alternating training scheme. The proposed technique shows improvement over its late and early fusion counterparts. Late fusion of different fusion techniques namely late, early and hybrid fusion shows state of the art results on Chalearn LAP cultural event recognition dataset.