Extraction of frame-difference features based on PCA and ICA for lip-reading

Abstract

The features of human lip motion from video clips are extracted by principal component analysis (PCA) and independent component analysis (ICA). Unlike many other features extracted from single-frame static images or multi-frame dynamic images, we extracted the features from the differences of consecutive frames. The PCA results in global features, while local features are extracted by the ICA. The features are extracted from several consecutive multi-frame differences as well as single-frame differences. The dynamic nature of multi-frame differences is more eminent. The resulting features maybe applicable in lip-reading and synthesis of lip motion videos with text-to-speech capability.