Uncovering the Origins of Instability in Dynamical Systems: How Can the Attention Mechanism Help?

Abstract

The behavior of the network and its stability are governed by both dynamics of the individual nodes, as well as their topological interconnections. The attention mechanism as an integral part of neural network models was initially designed for natural language processing (NLP) and, so far, has shown excellent performance in combining the dynamics of individual nodes and the coupling strengths between them within a network. Despite the undoubted impact of the attention mechanism, it is not yet clear why some nodes of a network obtain higher attention weights. To come up with more explainable solutions, we tried to look at the problem from a stability perspective. Based on stability theory, negative connections in a network can create feedback loops or other complex structures by allowing information to flow in the opposite direction. These structures play a critical role in the dynamics of a complex system and can contribute to abnormal synchronization, amplification, or suppression. We hypothesized that those nodes that are involved in organizing such structures could push the entire network into instability modes and therefore need more attention during analysis. To test this hypothesis, the attention mechanism, along with spectral and topological stability analyses, was performed on a real-world numerical problem, i.e., a linear Multi-Input Multi-Output state-space model of a piezoelectric tube actuator. The findings of our study suggest that the attention should be directed toward the collective behavior of imbalanced structures and polarity-driven structural instabilities within the network. The results demonstrated that the nodes receiving more attention cause more instability in the system. Our study provides a proof of concept to understand why perturbing some nodes of a network may cause dramatic changes in the network dynamics.