Optical Simulation and Image Quality for Automotive Semantic Segmentation

Abstract

Cameras are often deployed in automated driving systems for multiple perception tasks, including object recognition. In previous work, a novel optical simulation framework, Superposition of Optical Lenses in Automotive Simulation (SOLAS), was proposed to simulate wide-angle and fisheye cameras on automotive data for 2D object detection. The main focus of this work is to assess semantic segmentation networks in relation to camera quality, mainly used for collision avoidance, lane marking detection, and enhanced road navigation. Characterising semantic segmentation performance across the full range of camera production tolerances is essential, given that image quality may vary across units. This study further develops the novel optical simulation framework, SOLAS, where a Pythonic ray-tracing library, Kraken Optical Simulator (KrakenOS), is adapted to simulate wide-angle surround-view optical systems across a range of optical defocus, directly in automotive scenes. Optical defocus is achieved through sensor displacement. In this paper, we propose extending SOLAS with Automated Reference-Free Defocus characterisation for Automotive Near-Field Cameras (ARDÁN), a pipeline for measuring image quality in automotive scenes. Furthermore, we demonstrate that our framework, utilising a diverse set of cameras and dense grids of Point Spread Functions (Point Spread Functions (PSFs)), is scalable in semantic segmentation, achieving consistent results across various datasets supported by statistical metrics. The primary metric used for performance evaluation is mean Intersection over Union (mean Intersection over Union (mIoU)). Based on our results, the Segformer network with 24.73 M parameters was the most robust across wide-angle automotive datasets, namely, nutonomy Scenes (nuScenes) and Virtual Karlsruhe Institute and Toyota Technological Institute (KITTI) 2.0. In nuScenes, image sharpness $(cycles/pixel)$ dropped by 20.5% from a nominal focus, with a small Segformer performance drop from 55.21% mIoU by 0.37%. In Virtual KITTI 2.0, image sharpness dropped by 44.66% from a nominal focus, with a Segformer performance drop by 37.42%. However, for fisheye, the Proportional-Integral-Derivative (PID)Net model with 28.76 M parameters was the most robust model across the defocus range. In Woodscape, image sharpness dropped by 53.45% from a nominal focus, with a PIDNet performance drop by 8.55%. In Parallel Domain Woodscape, image sharpness dropped by 34.46% from a nominal focus, with a PIDNet performance drop by 7.28%.