VisualRWKV-HM: Enhancing linear visual-language models via hybrid mixing

Abstract

With the success of Large Language Models, Visual Language Models (VLMs) have also developed rapidly. However, existing VLMs often face limitations due to their quadratic time and space complexity, which poses challenges for training and deployment. Linear VLMs have emerged as a solution, providing linear time and space complexity, along with advantages in training and deployment. Nevertheless, a performance gap remains compared to state-of-the-art (SOTA) VLMs. This paper proposes VisualRWKV-HM, a model with linear complexity that incorporates a hybrid mixing mechanism combining time mixing and cross state mixing. This design achieves an optimal balance in information utilization, enhancing performance and offering flexibility for various tasks. VisualRWKV-HM achieves SOTA performance across single-image, multi-image, and multi-view benchmarks, and significantly outperforms the vanilla VisualRWKV. It demonstrates high computational efficiency with a context length of 24K, being 2.96 times faster and reducing memory usage by 45.38% compared to the Transformer-based LLaVA-1.5. When compared to LongLLaVA, a hybrid model based on the Transformer-Mamba architecture, it consumes less memory and achieves a 24% improvement in throughput at a context length of 16K. Additionally, we show that VisualRWKV-HM has strong scalability, with the potential for improved performance by scaling up the state encoder and decoder.