DeGMix: Efficient Multi-Task Dense Prediction With Deformable and Gating Mixer

Abstract

Convolution neural networks (CNNs) and Transformers offer complementary strengths for multi-task dense prediction: CNNs are effective at modeling local spatial patterns, while Transformers capture long-range dependencies. However, existing methods often rely primarily on one paradigm or do not fully exploit their complementarity for task-aware representation learning. In this work, we propose DeGMix (Deformable and Gating Mixer), a unified encoder–decoder framework for multi-task dense prediction that combines deformable convolution and attention-based task interaction. DeGMix consists of a deformable mixer encoder for task-aware spatial feature extraction and a task-aware gating Transformer decoder for cross-task interaction and task-specific feature selection. In this way, the proposed framework jointly models local deformable cues, global dependencies, and task-aware feature adaptation within a single architecture. Extensive experiments on three benchmark datasets, including NYUD-v2, PASCAL-Context, and Cityscapes, show that DeGMix consistently outperforms competitive CNN-based and Transformer-based multi-task methods across multiple dense prediction tasks. For example, with a Swin-L backbone, DeGMix achieves 57.55 mIoU on semantic segmentation on NYUD-v2, surpassing the previous best result by + 5.2 mIoU. In parameter-efficient fine-tuning on PASCAL-Context, DeGMix obtains a $+ 1.96~\Delta _{m}$ gain with 2.2M trainable parameters, approaching MTL-full fine-tuning ( $+ 2.23~\Delta _{m}$ ) while updating $13.7\times $ fewer parameters. Our code and models are publicly available at https://github.com/yangyangxu0/DeGMix