Transformer-based Multimodal Contextual Co-encoding for Humour Detection

Humor, a unique expression of the human language system different from other emotions, plays a very important role in human communication. Previous works on humor detection have been mostly limited to a single textual modality. From the perspective of human humor perception, various aspects such as text, intonation, mannerisms, and body language can convey humor. From the perspective of the structure of jokes, any combination of textual, acoustic, and visual modalities in various positions in the context can form unexpected humor. Therefore, information that exists among multiple modalities and contexts should be considered simultaneously in humor detection. This paper proposes a humor detection model based on the transformer and contextual co-encoding called Transformer-based Multimodal Contextual Co-encoding (TMCC). The model uses the transformer-based multi-head attention to capture potential information across modalities and contexts first. Then, it uses a convolutional autoencoder to further fuse the overall feature matrix and reduce dimensionality. Finally, a simple multilayer perceptron is used to predict the humor labels. By comparing with common baselines of humor detection, it is demonstrated that our model achieves some performance improvement. The availability of each part of the model is demonstrated through a series of ablation studies.