Ray carving with gradients and motion

Abstract

Recent developments in camera and computer technology have made multiple-camera systems less expensive and more usable. Using such systems, we can generate 3D models of human activity for use in surveillance, as avatars, or for 3D effects generation. Some approaches to model generation are voxel coloring, space carving, silhouette intersection, and the combination of multiple stereo reconstructions. Our attempt to overcome various shortcomings of the above approaches has led to the use of image derivatives and motion to determine the shape and motion of the activity in view. Direct computations of the gradient directions and the image motion normal to the gradient provide the information to generate a 3D+motion model that is consistent with all the image data. Data structures encode visibility information from each of the cameras surrounding the scene, allowing efficient determination of the subsets of measurements to be combined in a modified space-carving system. The main contributions of this paper are: the development of a system for combining multiple image gradient measurements to determine the 3D iso-brightness direction and its consistency, a system for combining multiple normal flow measurements to determine the motion normal to the iso-brightness direction, and a data structure based on the rays passing through the centers of projection and the image pixels, forming an unbounded projective grid through the space of the scene and allowing efficient determination and updating of scene point visibility. Reconstructions of human motion using 20 cameras are presented.