TensorProjection layer: A tensor-based dimension reduction method in deep neural networks

Abstract

In this study, we propose a dimension reduction method for features with tensor structure, implemented as a neural network layer called the TensorProjection Layer. This layer applies mode-wise linear projections to the input tensor to reduce its dimensionality, with the projection directions treated as trainable parameters optimized during model training.

The method is particularly useful for image data, serving as an alternative to pooling layers that reduce spatial redundancy. It can also reduce channel dimensions, making it applicable to various forms of tensor compression. While especially effective for image-based tasks, its application is not limited to them—as long as the intermediate representation is a tensor. We also demonstrate its use in multi-channel time-series and language data, showcasing its flexibility across diverse modalities.

We evaluate the method by replacing specific layers in standard baseline models with TPL, across tasks including medical image classification and segmentation, classification of medical time-series signals, and classification of medical abstract texts. Experimental results suggest that, compared to conventional downsampling techniques such as pooling, the proposed layer offers improved generalization performance, making it a promising alternative for feature summarization in diverse neural network architectures.