Inside-out Vision for Procedure Recognition in Dental Environment

Smart homes and offices are becoming more and more common with the advances in computer vision research and technology. Identifying the human activities and scenarios are basic components of such systems. This is important not only for the eco-system to work independently, but also to allow robots to be able to assist humans. This is specially true in the more complicated medical setups, e.g. dentistry, where we need subtle cues e.g. eye motion to identify scenarios. We present a hierarchical model in this paper for robustly recognizing scenarios and procedures in a dental setup by using the objects seen in eye gaze trajectories like material and equipment used by the dentist, and symptoms of the patient. We utilize the fact that by identifying the objects viewed during an activity and linking them over time to create more complicated scenarios, the problem of scenario recognition can be hierarchically solved. We performed experiments on a dental dataset and showed that combining multiple parameters results in a better precision and accuracy compared to any of them individually. Our experiments show that the accuracy increased from 45.18% to 94.42% when we used a combination of parameters vs. a single one.