FaithViP: A Benchmark for Hallucinations With Synthetic Visual Prompts in Large Multi-Modal Models

Abstract

Visual prompts serve as a representative type of synthetic objects, showing promise for guiding large multi-modal models (LMMs) to achieve precise region-specific understanding in applications like medical imaging and autonomous driving. However, the free synthetic form and pixel-level granularity of visual prompts make them a significant hallucination source. Despite existing benchmarks for general object hallucination, there is a critical need for a dedicated benchmark to evaluate the faithfulness of LMMs when faced with diverse synthetic prompts. In this paper, we introduce FaithViP, a comprehensive synthetic-visual-prompt benchmark to assess the faithfulness for human-interactive reasoning in LMMs. Drawing insights from human cognition, we design diverse hallucination scenarios, including missing or ambiguous visual prompts and object references, attribute conflicts between instruction and marked region. FaithViP comprises 308 meticulously crafted image-question pairs that induce visual-prompt hallucinations. We evaluate a wide range of LMM baselines, including general LMMs and those fine-tuned on visual-prompt data. Our results show that even SOTA LMMs like GPT-4o, Qwen2.5-VL and ViP-LLaVA series struggle to achieve optimal performance, showing the challenge of our FaithViP. Furthermore, we analyze the underlying causes of visual-prompt hallucinations, uncovering the limitations of existing open-source LMMs. Our work provides key insights into enhancing visual-textual interaction capabilities for LMMs, advancing more robust real-world applications. Our code and data are available at https://github.com/YFCodeDream/FaithViP