A general debiasing framework with counterfactual reasoning for multimodal public speaking anxiety detection

Abstract

Multimodal Public Speaking Anxiety Detection (MPSAD), which aims to identify the anxiety states of learners, has attracted widespread attention. Unfortunately, the current MPSAD task inevitably suffers from the impact of latent different types of multimodal hybrid biases, such as context bias, label bias and keyword bias. Models may rely on these biases as shortcuts, preventing them from fully utilizing all three modalities to learn multimodal knowledge. Existing methods primarily focus on addressing specific types of biases, but anticipating bias types when designing these methods is challenging, as we cannot foresee all possible biases. To tackle this issue, we propose a General Multimodal Counterfactual Reasoning debiasing framework (GMCR), which eliminates multimodal hybrid biases from a unified causal perspective. Specifically, this plug-and-play debiasing framework removes multimodal hybrid biases by disentangling causal and biased features and capturing adverse effects via a counterfactual branch. It then subtracts spurious correlations during inference for unbiased predictions. Due to the challenge of collecting speech video data, there are currently limited high-quality datasets available for the MPSAD task. To overcome this scarcity, we create a new large-scale fine-grained Multimodal English Public Speaking Anxiety (ME-PSA) dataset. Extensive experiments on our ME-PSA and two benchmarks demonstrate the superiority of our proposed framework, with improvements of over 2.00% in accuracy and 4.00% in F1 score compared to the vanilla SOTA baselines.¹