Artificial intelligence automated solution for hazard annotation and eye tracking in a simulated environment

High-fidelity simulators and sensors are commonly used in research to create immersive environments for studying real-world problems. This setup records detailed data, generating large datasets. In driving research, a full-scale car model repurposed as a driving simulator allows human subjects to navigate realistic driving scenarios. Data from these experiments are collected in raw form, requiring extensive manual annotation of roadway elements such as hazards and distractions. This process is often time-consuming, labor-intensive, and repetitive, causing delays in research progress.

This paper proposes an AI-driven solution to automate these tasks, enabling researchers to focus on analysis and advance their studies efficiently. The solution builds on previous driving behavior research using a high-fidelity full-cab simulator equipped with gaze-tracking cameras. It extends the capabilities of the earlier system described in Pawar’s (2021) “Hazard Detection in Driving Simulation using Deep Learning”, which performed only hazard detection. The enhanced system now integrates both hazard annotation and gaze-tracking data.

By combining vehicle handling parameters with drivers’ visual attention data, the proposed method provides a unified, detailed view of participants’ driving behavior across various simulated scenarios. This approach streamlines data analysis, accelerates research timelines, and enhances understanding of driving behavior.