Heuristic Once Learning for Image & Text Duality Information Processing

Abstract

Few-shot learning is an important mechanism to minimize the need for the labeling of large amounts of data and taking advantage of transfer learning. To identify image/text input with duality property, this research proposes a “Heuristic once learning (HOL)” mechanism to investigate multi-modal input processing similar to human-like behavior. First, we create an image/text data set of big Latin letters composed of small letters and another data set composed of Arabic, Chinese and Roman numerals. Secondly, we use Convolutional Neural Networks (CNN) for pre-training the dataset of letters to get structural features. Thirdly, using the acquired knowledge, a Self-organizing Map (SOM) and Contrastive Language-Image Pretraining (CLIP) are tested separately using zero-shot learning. Siamese Networks and Vision Transformer (ViT) are also tested using one-shot learning by knowledge transfer to identify the features of unknown characters. The research results show the potential and challenges to realize HOL and make a useful attempt for the development of general agents.