Towards Reliable Multimodal Intelligence via Uncertainty-Aware Inference

Abstract

Large vision-language models (LVLMs) offer transformative potential for multimodal reasoning in intelligent transportation systems. However, their deployment in safety-critical scenarios is hindered by overconfident predictions, computationally intensive uncertainty estimation, and unreliable confidence calibration. Existing methods, such as semantic entropy and Monte Carlo dropout, require multiple forward passes, rendering them impractical for billion-scale models. Moreover, current cross-modal retrieval approaches often neglect uncertainty in vision-language alignment. To address these challenges, we introduce a unified framework comprising three key innovations: Uncertainty-aware quadruple multimodal contrastive learning with constraints and self-penalization (UQUEST), an uncertainty-aware retrieval module that combines learnable query-based aleatoric uncertainty with dual decoders to enhance both retrieval and generation; vsion-language fast layer-wise uncertainty estimation (VL-FLUE), a layer-wise uncertainty estimation technique that exploits transformer redundancy for efficient single-pass inference; and vision-language adaptive temperature scling, a context-aware calibration method based on adaptive temperature scaling. Extensive evaluations on Microsoft common objects in context, Flickr30K, and DrivingVQA demonstrate notable gains: UQUEST enhances retrieval-augmented generation performance, while VL-FLUE reduces inference cost and improves calibration quality. This framework advances the reliability and efficiency of uncertainty quantification, paving the way for trustworthy LVLM deployment in real-time transportation applications.