From words to visuals: Bridging text and visual insights using MetA-MARC framework for enhanced scholarly article categorization

Abstract

The rapid growth of technology has led to approximately 28,100 journals disseminating 2.5 million research articles annually, posing significant challenges in locating and categorizing articles of interest. Search engines, citation indexes, and digital libraries often return predominantly irrelevant papers due to limited indexing. Existing classification techniques leveraging content and metadata face challenges such as incomplete data and lack of semantic context. Metadata-based methods frequently rely on statistical metrics that neglect semantic meanings and require subject expertise for threshold setting. To address these issues, we propose Metadata-Driven Attention-Based Multimodal Academic Research Classifier (MetA-MARC), a framework leveraging the pretrained CLIP model to integrate text and image modalities for enhanced scholarly article classification. MetA-MARC captures semantic and contextual meaning by integrating metadata, OCR-extracted features, and images through CLIP (Contrastive Language-Image Pre-training). It introduces a novel textual inversion approach to map images to pseudo-word tokens in the CLIP embedding space for robust multimodal representations. The framework employs FusionWeave, a multimodal fusion network combining features using concatenation, cross fusion, and attention-based techniques, alongside Modality-Driven Adaptive Re-weighting (MoDAR) to dynamically prioritize relevant features. Experiments on JUCS, ACM, and proprietary CompScholar datasets demonstrate average accuracies of 0.86, 0.84, and 0.8848, respectively, surpassing state-of-the-art methods by up to 4.05%. These results highlight MetA-MARC’s potential as a robust, adaptive tool for automated scholarly article classification, effectively bridging text and visual modalities.