Towards the future of pedestrian–AV interaction: Human perception vs. LLM insights on Smart Pole Interaction Unit in shared spaces

Abstract

As autonomous vehicles (AVs) reshape urban mobility, establishing effective communication between pedestrians and self-driving vehicles has become a critical safety imperative. This work investigates the integration of Smart Pole Interaction Units (SPIUs) as external human–machine interfaces (eHMIs) in shared spaces and introduces an innovative approach to enhance pedestrian–AV interactions. To provide subjective evidence on SPIU usability, we conduct a group design study (“Humans”) involving 25 participants (aged 18–40). We evaluate user preferences and interaction patterns using group discussion materials, revealing that 90% of the participants strongly prefer real-time multi-AV interactions facilitated by SPIU over conventional eHMI systems, where a pedestrian must look at multiple AVs individually. Furthermore, they emphasize inclusive design through multi-sensory communication channels—visual, auditory, and tactile signals—specifically addressing the needs of vulnerable road users (VRUs), including those with impairments. To complement these non-expert, real-world insights, we employ three leading Large Language Models (LLMs) (ChatGPT-4, Gemini-Pro, and Claude 3.5 Sonnet) as “experts” due to their extensive training data. Using the advantages of the multimodal vision-language processing capabilities of these LLMs, identical questions (text and images) used in human discussions are posed to generate text responses for pedestrian–AV interaction scenarios. Responses generated from LLMs and recorded conversations from human group discussions are used to extract the most frequent words. A keyword frequency analysis from both humans and LLMs is performed with three categories, Context, Safety, and Important. Our findings indicate that LLMs employ safety-related keywords 30% more frequently than human participants, suggesting a more structured, safety-centric approach. Among LLMs, ChatGPT-4 demonstrates superior response latency, Claude shows a closer alignment with human responses, and Gemini-Pro provides structured and contextually relevant insights. Our results from “Humans” and “LLMs” establish SPIU as a promising system for facilitating trust-building and safety-ensuring interactions among pedestrians, AVs, and delivery robots. Integrating diverse stakeholder feedback, we propose a prototype SPIU design to advance pedestrian–AV interactions in shared urban spaces, positioning SPIU as crucial infrastructure hubs for safe and trustworthy navigation.