Hybrid Camera System for Telepresence with Foveated Imaging

To improve the telepresence sense of a local HMD user, a high-resolution view of the remote environment is necessary. However, current commodity omnidirectional camera could not support enough resolution for the human eye. Using a higher resolution omnidirectional camera is also infeasible because it will increase the streaming bandwidth. We propose a hybrid camera system that can convey a higher resolution for the HMD user viewport ROI region in available bandwidth range. The hybrid camera consists of a pair of omnidirectional and PTZ camera which is close to each other. The HMD user head orientation controls the PTZ camera orientation. The HMD user also controls the zooming level of the PTZ camera to achieve higher resolution up to PTZ camera maximum optical zoom level. The remote environment view obtained from each camera is streamed to the HMD user and then stitched into one combined view. This combined view simulates human visual system (HVS) phenomenon called foveation, where only a small part in the human view is in high resolution, and the rests are in low resolution.