DSA-PR: Discrete Soft Biometric Attribute-Based Person Retrieval in Surveillance Videos

Abstract

Physical characteristics or soft biometrics are visually perceptible aspects of a human body. Noticeable attributes like build, height, complexion, clothes help with the development of a human surveillance system. The paper proposes Discrete Soft biometric Attribute-based Person Retrieval (DSA-PR) from a video using height, gender, torso (clothes) color-1, torso color-2, and torso (clothes) type given in a textual query. The DSA-PR uses Mask R-CNN for semantic segmentation and ResNet-50 for attribute classification. Height is estimated using the Tsai camera calibration method. DSA-PR weighs attributes and fuses their probability to generate a final score for each detected person. The proposed approach achieves an average Intersection-over-Union (IoU) of 0.602 and retrieval with IoU $\ge$ 0.4 is 0.808 over the AVSS challenge II dataset which works out to 5.8% and 2.02% above the state-of-the-art techniques respectively.