MedViA: Empowering medical time series classification with vision augmentation and multimodal fusion

Abstract

The analysis of medical time series, such as Electrocardiography (ECG) and Electroencephalography (EEG), is fundamental to clinical diagnostics and patient monitoring. Accurate and automated classification of these signals can facilitate early disease detection and personalized treatment, thereby improving patient outcomes. Although deep learning models are widely adopted, they mainly process signals as sequential numerical data. Such a single-modality approach often misses the holistic visual patterns easily recognized by clinicians from graphical charts and struggles to model the complex non-linear dynamics of physiological data. As a result, the rich diagnostic cues contained in visual representations remain largely untapped, limiting model performance. To address these limitations, we propose MedViA, a novel multimodal learning framework that empowers Medical time series classification by integrating both Vision Augmentation and numeric perception. Our core innovation is to augment the raw medical time series signals into the visual modality, enabling a dual-pathway architecture that computationally mimics the comprehensive reasoning of clinical experts. With the augmentation, MedViA then features two parallel perception branches: a Visual Perception Module, built upon a novel Multi-resolution Differential Vision Transformer, processes the augmented images to capture high-level structural patterns and diagnostically critical waveform morphologies. Concurrently, a Numeric Perception Module, which uses our proposed Temporal Kolmogorov Network to model fine-grained and non-linear dynamics directly from the raw time series. To synergistically integrate the insights from these dedicated pathways, we introduce a Medically-informed Hierarchical Multimodal Fusion strategy, which uses a late-fusion architecture and a hierarchical optimization objective to derive for the final classification. We have conducted extensive experiments on multiple public medical time series datasets, which demonstrate the superior performance of our method compared to state-of-the-art approaches.