Brain–Machine Cross-Modal Enhancement Method With Deep Supervision for Visual Classification

Abstract

Visual classification has achieved remarkable progress with the development of deep neural networks (DNNs). However, the reliance on large-scale labeled training sets limits the generalized classification ability of such models. To address this challenge, we propose a brain–machine cross-modal enhancement framework that leverages brain signals as auxiliary supervision during training. Unlike existing methods that impose brain signal constraints only at the final classification layer, our method introduces deep guidance through hidden-layer constraints, enabling better feature discrimination and improved knowledge transfer. Specifically, we adopt contrastive learning to align brain signal and image modalities in a shared embedding space, where brain signal-image pairs serve as cross-modal supervision. The image encoder is partitioned into multiple shallow components, each equipped with a dedicated classifier trained using soft targets distilled from the brain signal-image coupling space. This full-process guidance design facilitates the extraction of semantically rich features across layers and enhances model generalization. Experimental results demonstrate that our method outperforms previous brain–machine enhancement approaches. Visualization results further reveal that our method enables the visual model to focus on semantically relevant regions in a human-like manner. These findings demonstrate that brain signals can effectively enhance visual learning, offering a new perspective for brain-inspired artificial intelligence.