Viewing the 360° Future: Trade-Off Between User Field-of-View Prediction, Network Bandwidth, and Delay

Abstract

Predicting a user’s field-of-view (FoV) accurately can help to significantly reduce the high bandwidth requirements for 360° video streaming, as it enables sending only the tiles corresponding to the predicted FoV. Since many approaches for user head-orientation (i.e., FoV) prediction have been proposed in the literature, ranging from simple linear regression to more complex neural networks, it is difficult to comprehensively decide which method to use. Towards resolving this gap in knowledge, in this work we benchmark user prediction algorithms over an aggregation of multiple datasets and study the implications of this analysis. Our results demonstrate that it is indeed difficult for any prediction algorithm to accurately predict a user’s FoV beyond a very short future time window of approximately 300 ms. We also observe that users’ viewing behavior is dominated by sideways head movement, rather than up-and-down. These findings have implications on network bandwidth, latency, and playback buffering at the client: (1) Extra "padding" tiles are needed around the user’s FoV in order to correct for prediction errors; in particular, a rectangular padding achieves lower stall rate than square padding, for the same bandwidth usage; (2) Video playout buffers, network delay, and jitter need to be small in order to avoid stale predictions of the user’s field-of-view, which are only valid 300 ms into the future; (3) Per-video and per-user personalization of the padding can save bandwidth for slow-moving users or videos. We mathematically quantify these tradeoffs and present simulation results to demonstrate these findings and implications. Our results have implications for FoV prediction methods in future 360° streaming systems.