Multimodal Sensory Representation for Object Classification via Neocortically-inspired Algorithm

Abstract

This study reports our initial work on multimodal sensory representation for object classification. To form a sensory representation we used the spatial pooling phase of the Hierarchical Temporal Memory - a Neocortically-inspired algorithm. The classification task was carried out on the Washington RGB-D dataset in which the employed method provides extraction of non-hand engineered representations (or features) from different modalities which are pixel values (RGB) and depth (D) information. These representations, both early and lately fused, were used as inputs to a machine learning algorithm to perform object classification. The obtained results show that using multimodal representations significantly improve (by 5 %) the classification performance compared to a when a single modality is used. The results also indicate that the performed method is effective for multimodal learning and different sensory modalities are complementary for the object classification. Therefore, we envision that this method can be employed for object concept formation that requires multiple sensory information to execute cognitive tasks.