CAS-TJ: Channel attention shuffle and temporal jigsaw for audio classification

Abstract

Audio classification, including speech emotion, has been a topic of extensive research and applies to various virtual assistants and intelligent systems. Previous methods relied on handcrafted features such as spectrograms, but these features often have limitations due to their manual nature. Recently, hybrid models that use both end-to-end learning from raw audio with CNNs and Transformers have been developed to address this issue. However, when raw audio features are compressed through convolutional neural networks (CNNs), numerous channels are created, leading to redundancy or irrelevant information, while Transformers also have their limitations. Therefore, we propose Channel Attention Shuffle and Temporal Jigsaw (CAS-TJ) to generate more effective features and improve robustness. CAS divides channels into groups, multiplies them by attention weights, aggregates, and shuffles them. This process allows information to be exchanged among various channels, creating more discriminative channels. TJ generates frame patches of a specific size and uses mixing and matching during the learning process. This helps to better understand temporal relationships and detect discriminative patterns. Finally, we conduct experiments on the ESC-50 and Urban-8k datasets and find that the overall performance of CAS-TJ is better than the baseline models.