Multi-scale feature extraction and aggregation network for electroencephalography classification in face photo-sketch recognition task.

Face photo-sketch recognition task plays a crucial role in forensic investigation, human visual perception, and facial biometrics applications. The substantial modality gap between photographs and sketches, compounded by the influence of the semantic gap, poses a formidable challenge to recognition tasks. This study aims to propose an effective electroencephalography (EEG)-based approach to bridge this gap. In this paper, we introduce a face photo-sketch recognition paradigm (FPSR), a rapid serial visual presentation (RSVP) paradigm for the matching of face sketches. Based on this paradigm, we further proposed a new EEG signal feature decoding method called multi-scale feature extraction and aggregation network (MFEA). This network extracts shallow features in three dimensions and reconstructs three dimensional abstract features. Subsequently, the shallow features are aggregated with the deeper features to enhance the retention of all effective EEG signal features. These combined features are then input into the spatial module for specific dimensionality reduction. Experiments were conducted on one public and one self-conducted EEG RSVP datasets to evaluate the performance of our proposed MFEA. The experimental results demonstrate that, compared to previous methods, our MFEA exhibits superior performance in the EEG classification task.