Learned representation normalization: attention focusing with multiple input modules

A large, multi-modular neural network can be envisaged for use in a complex, multi-task application. The optimum data representation for each sub-task of such an application is often unknown and different from the optimum data representation for the other sub-tasks. A method is needed that allows a network that contains several alternate input representations to learn to focus its attention on the best representation(s) for each sub-task to be learned, without a priori information on best representation-sub-task combinations. An adaptive attention focusing method is introduced that addresses this issue. The method involves training recurrent connections for each input module to selectively attenuate input to that module that causes training error in a final target module. The method is shown to have similarities with both gating networks and anti-Hebbian learning. A task scenario is proposed for which adaptive attention focusing provides superior classification performance relative to standard training methods.<>