A complete in-cabin monitoring framework for autonomous vehicles in public transportation

Autonomous vehicles (AVs), driven by state-of-the-art deep learning and computer vision technologies, can revolutionize current mobility systems in modern transportation. Driverless AVs are slowly integrated into public transportation with significant advantages for the passengers and public transport operators. However, passenger safety and comfort are two of the main challenges that need to be addressed. This work presents a complete in-cabin monitoring framework with a suite of services, employing deep learning algorithms using a variety of onboard sensors at the edge. This proposed framework offers various innovative services aimed at enhancing security, monitoring passenger presence, accommodating diverse needs, and personalizing the passengers' travel experience, while also reducing the workload of human safety officers. Experimental results demonstrate the framework's effectiveness in identifying abnormal events with a high accuracy, employing multiple datasets and custom in-cabin scenarios. Additionally, the system effectively conducts automated passenger counting and facial identification, ensuring real-time responsiveness under diverse operational conditions. Overall, the novelty of this work lies in the framework's multimodal approach, integrating visual and audio analysis, to achieve robust performance across various scenarios, significantly contributing to the advancement of autonomous driving technologies.