Leveraging VVC partitioning for low-latency video decoder output

Abstract

Emerging video applications, such as cloud gaming and cloud rendering, rely on time critical decoding operations so that user interactions are reflected on the user's screen as soon as possible. In these applications content is encoded, transmitted and decoded before finally being displayed on the user's screen with each of these steps adding latency. Furthermore, often between the output of a decoded frame and its display additional operations may be required, such as up/downscaling, UI overlays or color space conversions. Current decoders only release the decoded samples once a full frame has been decoded, although under such low delay requirements, it would be beneficial if applications could start to process the decoded samples prior to the complete decoding of a coded picture. This would allow for parallel processing of a frame in the decoder and throughout the display pipeline, which could ultimately allow for lower latencies and eventually pave the way for more enhancement operations without inducing latency. We modified a decoder for the newly published Versatile Video Coding (VVC) codec to demonstrate that it is possible to output partial frames. The changes in the decoder to enable such partial output are straightforward and allow for the new partitioning schemes to be leveraged for partial output. The demonstrator will show how the progressive output can then be used by elements further on in the decoding/display pipeline.