Toward Automated and Comprehensive Walkability Audits with Street View Images: Leveraging Virtual Reality for Enhanced Semantic Segmentation

Abstract

Street view images (SVIs) coupled with computer vision (CV) techniques have become powerful tools in the planning and related fields for measuring the built environment. However, this methodology is often challenging to be implemented due to challenges in capturing a comprehensive set of planning-relevant environmental attributes and ensuring adequate accuracy. The shortcomings arise primarily from the annotation policies of the existing benchmark datasets used to train CV models, which are not specifically tailored to fit urban planning needs. For example, CV models trained on these existing datasets can only capture a very limited subset of the environmental features included in walkability audit tools. To address this gap, this study develops a virtual reality (VR) based benchmark dataset specifically tailored for measuring walkability with CV models. Our aim is to demonstrate that combining VR-based data with the real-world dataset (i.e., ADE20K) improves performance in automated walkability audits. Specifically, we investigate whether VR-based data enables CV models to audit a broader range of walkability-related objects (i.e., comprehensiveness) and to assess objects with enhanced accuracy (i.e., accuracy). In result, the integrated model achieves a pixel accuracy (PA) of 0.964 and an intersection-over-union (IoU) of 0.679, compared to a pixel accuracy of 0.959 and an IoU of 0.605 for the real-only model. Additionally, a model trained solely on virtual data, incorporating classes absent from the original dataset (i.e., bollards), attains a PA of 0.979 and an IoU of 0.676. These findings allow planners to adapt CV and SVI techniques for more planning-relevant purposes, such as accurately and comprehensively measuring walkability.