In situ spatial AR surgical planning using projector-Kinect system

Traditional surgical planning is challenged by limited spatial information of 2-D medical images or separated planning phases between 3-D model based virtual planning and its in situ registration. This paper presents an augmented surgical planning method with model-section views of an actual patient body and direct augmented interaction to realize spatially interactive planning in situ. A projector-Kinect system is proposed to construct a spatially augmented reality surgical environment directly on the patient body through surgical models' projection, correction and registration. Hand gesture based direct augmented interaction is developed to provide surgeons with interactive guidance during planning. The physical patient's body is augmented by the integration of medical information like 3-D anatomic and pathologic models as well as the 2-D medical images. Compared to conventional planning methods, this augmented planning enables surgical planning in 3-D space on the actual patient, and allows the planning to be integrated into an in situ robot-assisted surgical environment. Experimental results demonstrate that the method can facilitate spatial information perception and convey virtual planning on the physical patient in situ.