An improved human-in-the-loop model for fine-grained object recognition with batch-based question answering

Abstract

Fine-grained object recognition refers to a subordinate level of object recognition such as recognition of bird species and car models. It has become crucial for recognition of previously unknown classes. While fine-grained object recognition has seen unprecedented progress with the advent of neural networks, many of the existing works are cost-sensitive as they are acutely picture-dependent and fail without the adequate number of quality pictures. Efforts have been made in the literature for a picture-independent recognition with hybrid human-computer recognition methods via single question answering with a human-in-the-loop. To this end, we propose an improved batch-based local question answering method for making the recognition efficient and picture-independent. When pictures are unavailable, at each time-step, the proposed method mines a batch of binary cluster-centric local questions to pose to a human-in-the-loop and incorporates the responses received to the questions into the model. After a preset number of time-steps, the most probable class of the target object is returned as the final prediction. When pictures are available, our model facilitates the plug-in of computer vision algorithms into the framework for better performance. Experiments on three challenging datasets show significant performance improvement with respect to accuracy and computation time as compared to the existing schemes.