Traffic scene perception via multimodal large language model with data augmentation and efficient training strategy

Abstract

Intelligent mobility, driven by advancements in deep learning and computing power, enhances transportation efficiency and societal connectivity, fostering economic and urban development. Current computer vision solutions often struggle to capture the complex details or understand the context within traffic scenes, limiting advanced intelligent mobility and raising safety concerns. Multimodal Large Language Models (MLLMs), by integrating linguistic and visual data, can aid vehicles and transportation systems in gaining a deeper understanding of the real-world traffic scenes, offering solutions to current challenges. Nevertheless, existing approaches predominantly employ MLLMs as instruments for querying and engaging with traffic infrastructure, rather than empowering these models to genuinely comprehend the traffic environment. This limitation curtails the potential of MLLMs and may even pose safety risks. In this paper, we first introduce a data augmentation framework designed to transform raw data into datasets suited for specific training objectives, thereby addressing issues related to data scarcity. Secondly, we propose a learning rate-based staged training strategy that segments the training process into distinct stages. This strategy involves deploying datasets targeted at various training objectives according to the patterns of parameter changes observed in different stages, thereby enhancing the training efficiency of the model. Utilizing these methods, we present InsightGPT, a model endowed with robust understanding and reasoning capabilities in traffic scenarios. In experiments conducted across six tasks, InsightGPT consistently outperforms baseline MLLMs in evaluating both the overall traffic scenes and individual objects within it, demonstrating its superior traffic comprehension and reasoning abilities. InsightGPT’s parameters and deployment details are available at https://github.com/JinleLiu/InsightGPT.