BodyExplorerAR: enhancing a mannequin medical simulator with sensing and projective augmented reality for exploring dynamic anatomy and physiology

Abstract

BodyExplorerAR is a system designed to enhance a learner's ability to explore anatomy, physiology and clinical interventions though naturalistic interaction with an augmented reality enhanced full-body mannequin simulator. We are developing a platform that integrates projective AR and multi-modal sensor inputs. A user can use an IR pen to open, resize and move viewports providing windows into the body that can display dynamic anatomy. The user can point to an organ and display additional information such as graphs of physiological parameters or heart sounds. Custom sensing systems provide natural interactions with common medical devices such as syringes, breathing tubes and catheters. A user can open a window displaying the beating heart in situ, display an electrocardiogram (ECG), then inject drugs and see and hear changes in heart rate. Our goal is an engaging experience that empowers a learner to create customized, media-rich explorations revealing the internal consequences of external actions.