Image Systems Simulation for 360° Camera Rigs

Camera arrays are used to acquire the 360◦ surround video data presented on 3D immersive displays. The design of these arrays involves a large number of decisions ranging from the placement and orientation of the cameras to the choice of lenses and sensors. We implemented an open-source software environment (iset360) to support engineers designing and evaluating camera arrays for virtual and augmented reality applications. The software uses physically based ray tracing to simulate a 3D virtual spectral scene and traces these rays through multi-element spherical lenses to calculate the irradiance at the imaging sensor. The software then simulates imaging sensors to predict the captured images. The sensor data can be processed to produce the stereo and monoscopic 360◦ panoramas commonly used in virtual reality applications. By simulating the entire capture pipeline, we can visualize how changes in the system components influence the system performance. We demonstrate the use of the software by simulating a variety of different camera rigs, including the Facebook Surround360, the GoPro Odyssey, the GoPro Omni, and the Samsung Gear 360. Introduction Head mounted visual displays can provide a compelling and immersive experience of a three-dimensional scene. Because the experience can be very impactful, there is a great deal of interest in developing applications ranging from clinical medicine, behavioral change, entertainment, education and experience-sharing [1] [2]. In some applications, computer graphics is used to generate content, providing a realistic, but not real, experience (e.g., video games). In other applications, the content is acquired from a real event (e.g., sports, concerts, news, or family gathering) using camera arrays (rigs) and subsequent extensive image processing that capture and render the environment (Figure 1). The design of these rigs involves many different engineering decisions, including the selection of lenses, sensors, and camera positions. In addition to the rig, there are many choices of how to store and process the acquired content. For example, data from multiple cameras are often transformed into a stereo pair of 360◦ panoramas [3] by stitching together images captured by multiple cameras. Based on the user’s head position and orientation, data are extracted from the panorama and rendered on a head mounted display. There is no single quality-limiting element of this system, and moreover, interactions between the hardware and software design choices limit how well metrics of individual components predict overall system quality. To create a good experience, we must be able to assess the combination of hardware and software components that comprise the entire system. Building and testing a complete rig is costly and slow; hence, it can be useful to obtain guidance about design choices by using Figure 1. Overview of the hardware and software components that combine in an camera rig for an immersive head-mounted display application. (A) The simulation includes a 3D spectral scene, the camera rig definition, and the individual camera specifications. This simulation produces a set of image outputs. (B) The images are then processed by a series of software algorithms. In this case, we show a pipeline that produces an intermediate panorama representation and the viewport calculations that render an image dependent on the users head position. a simulation of the system. This paper describes software tools that simulate controlled 3D realistic scenes and image acquisition systems, in order to generate images produced by specific hardware choices. These images are the inputs to the stitching and rendering algorithms. The simulation enables engineers to explore the impact of different design choices on the entire imaging system, including realistic scenes, hardware components, and post-processing algorithms. Software Implementation The iset360 software, which models the image capture pipeline of 360 camera rigs, has portions in MATLAB and portions in C++. The simulation software is freely available in three repositories within the ISET GitHub project: https://github.com/ISET 1. Figure 2 and Figure 3 summarize the initial stages of the workflow. The first portion of the code creates realistic 3D scenes and calculates the expected sensor irradiance given a lens description. To do so, we start with a 3D, virtual scene that is constructed using 3D modeling software (e.g. Blender or Maya). The scene is converted into a format compatible with PBRT [4], which is implemented in C++. PBRT is a quantitative computer graphics tool that we use to calculate the irradiance at the sensor as light travels from the 3D scene, through the lens, and onto the sensor surface. We augmented the PBRT code to return multispectral images, model lens diffraction and simulate light fields [5]. To promote platform in1The three repositories are iset360, iset3d, and isetcam