Design of audio to image cross-modal learning and generation based on single-layer CoPt spin-orbit torque devices

Abstract

Despite remarkable achievements in neural network applications using spin–orbit torque (SOT) devices, current research predominantly focuses on unimodal classification tasks, while cross-modal learning tasks often suffer from suboptimal generation quality. To address this limitation, we employ generative adversarial networks (GANs) based on CoPt-SOT to achieve cross-modal learning and generation from speech to handwritten digit images. Initially, we explore the field-free switching characteristics of single-layer CoPt devices, developing spin-based implementations of Scaled Rectified Linear Unit (SReLu), Sigmoid, and Tanh neuronal functions. Simultaneously, we leverage the nonvolatile multistate characteristics of CoPt devices to realize excitatory-inhibitory synaptic plasticity. Subsequently, for the first time, we construct half-spin and full-spin GAN networks using these spin-based neurons and synapses, enabling cross-modal learning and generation between speech and image domains. Finally, we evaluate the performance of these networks through handwritten digit recognition tasks, achieving impressive recognition accuracies of 93.78 % and 88.61 % for half-spin and full-spin implementations, respectively. Notably, this work overcomes the existing bottleneck in SOT device applications, which have been largely confined to unimodal classification tasks, and demonstrates significant potential for expanding the scope of SOT-based technologies.