CM-CIF: Cross-Modal for Unaligned Modality Fusion with Continuous Integrate-and-Fire

Abstract

The purpose of Audio-Visual Speech Recognition is to identify the content of the spoken sentence by extracting the lip movement features and acoustic features from an input video file containing a person's conversation. Although the current audio-visual fusion models solve the problem of inconsistency in the time length of different modalities to a certain extent, the fusion of the modalities may cause acoustic boundary ambiguity. To better solve this problem, in this paper, we propose a model named Cross-Modal Continuous Integrate-and-Fire (CM-CIF). The model integrates cross-modal information to the accumulated weight so that the acoustic boundary can be located more accurate. We use the Transformer-seq2seq model as the baseline and test CM-CIF on the public datasets LRS2 and LRS3. Experimental results show that CM-CIF achieves a competitive performance.