A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Models

Abstract

Visual token compression is critical for Large Vision-Language Models (LVLMs) to efficiently process high-resolution inputs. Existing methods that typically adopt fixed compression ratios cannot adapt to scenes of varying complexity, often causing imprecise pruning that discards informative visual tokens and results in degraded model performance. To address this issue, we introduce a dynamic pruning framework, GlimpsePrune, inspired by human cognition. It takes a data-driven “glimpse” and prunes irrelevant visual tokens in a single forward pass before answer generation. This approach prunes 92.4% of visual tokens while on average nearly preserving the original performance on free-form VQA tasks, causing end-to-end latency and peak memory to reduce to 66% and 67% of the baseline, respectively. By integrating our pruning mechanism into the reinforcement learning fine-tuning stage, the time and memory costs of fine-tuning are also reduced with negligible performance loss. Code is available at https://github.com/HVision-NKU/GlimpsePrune