Sign Words Annotation Assistance Using Japanese Sign Language Words Recognition

Abstract

A Japanese sign language corpus is essential to activate analysis and recognition research of Japanese sign language. It requires collecting large scale of video data and annotating information to build a sign language corpus. Generally, building a sign language corpus is tedious work, and assistance is necessary. This paper describes one of the assistance methods for annotation tasks of sign words using Japanese sign language words recognition. The words recognition extracts sign features from a video, segments it into meaningful units, and annotates word labels to them automatically. At this time, the user's annotation tasks can be reduced from the full-manual work to confirmation and correction of the annotation. The proposed sign words recognition is composed of body-parts tracking, feature extraction, and words classification. The five types of approaches including i) feature fusion and ii) multi-stream HMM to handle the multiple body-parts are applied and compared. We build a video database of Japanese sign language words and a manual annotation interface to evaluate the proposed method. The database includes 92 Japanese sign language words which are signed by ten native signers. The total number of videos is 4,590, and 3,900 videos of 78 words except for recording and sign errors are used for the evaluation. The classification accuracies were 75.88% and 93.35% in the signer and trial opened conditions, respectively, when the parts-based feature fusion and multi-stream HMM using relative weights for body-parts are employed. Moreover, the expected work reduction ratio of annotation tasks using the interface was 38.01%.