Learning generic and specific prompts with contrastive constraints for multi-task visual scene understanding

Multi-task learning has emerged as a crucial research direction in the field of computer vision, offering improved performance and efficiency across multiple tasks. Recent studies have incorporated prompt learning into multi-task learning to enhance the interaction between prompt vectors and image representations. However, these studies fail to consider the inter-task and intra-task relations of prompt vectors under multi-task scenarios. To address this issue, we propose learning Generic and Specific Prompts (GSPrompt) with contrastive constraints for multi-task visual scene understanding. Our approach assumes that each task possesses both commonality and individuality, leading us to design two distinct types of prompt vectors: task-generic prompts and task-specific prompts. By constraining the prompt vectors through pulling task-generic prompts and pushing task-specific prompts, we enable multi-task models to learn prompts capable of adapting to multiple tasks simultaneously. Extensive experiments on NYUD-v2, PASCAL-Context, and Cityscapes show that GSPrompt learns effective prompts and achieves state-of-the-art performance. The code is publicly available at https://github.com/teeyohan/GSPrompt-main.