Spotting fingerspelled words from sign language video by temporally regularized canonical component analysis

A method for spotting specific words in sign language video is proposed. In classes and talks given using Japanese Sign Language, words that do not have a defined sign, such as the names of people, objects, and places, are represented by sets of multiple characters from the Japanese finger alphabet. The difficulty of recognizing these words has created strong demand for the ability to spot specific words in order to help interpreters and the audience to follow a talk. We address the spotting task by employing the basic idea of temporal regularized canonical correlation analysis (TRCCA), which can simultaneously handle shape and motion information about a 3D object. The classification accuracy of TRCCA is enhanced by incorporating two functions: 1) parallel processing with multiple time scales, 2) strong implicit feature mapping by nonlinear orthogonalization. The enhanced TRCCA is called "kernel orthogonal TRCCA (KOTRCCA)". The effectiveness of the proposed method using KOTRCCA is demonstrated through experiments spotting eight different words in sign language videos.