Multimodal tracking for smart videoconferencing

Many interactive multimedia applications require the ability to track the 3-D motion of participants in a room. Particle filters are attractive for this since they do not require solution of the inverse problem of obtaining the state from measurements, and since the tracking can be easily extended to integrate multimodal measurements. We extend our previous work on smart videoconferencing to include a multimodal tracker of the session participants using multiple cameras and microphone arrays. We verify the correctness and robustness of the multimodal tracker using synthetic and real data. We also present practical details of how such a system can be implemented using off-the-shelf hardware and computers.