Lip detection and tracking

Abstract

Seeing the talker's lips in addition to audition can improve speech understanding which is rather based on lip shape temporal evolution than on absolute mouth shape. We propose a totally automatic algorithm which can extract lip shape over an image sequence. The algorithm does not require any make-up or markers and works under natural lighting conditions. The lip detection algorithm uses an active shape model to describe the mouth. After a training step, the mouth model is iteratively deformed under constraints according to spatiotemporal energies. The robust prior detection of mouth corners and Cupidon's arch yields the automatic positioning of the initial shape which is very difficult and must be as accurate as possible. Temporal information integration comes from the definition of Kalman filters on the independent mouth parameters. Such filtering gives an initial shape close to the final one which speeds up the convergence rate. We point out on the behaviour of our algorithm when a transition open mouth/closed mouth or closed mouth/open mouth occurs.