Liquid and solid layers in a thermal deep-learning machine.

Abstract

We study generalization in overparameterized deep neural networks using a thermal deep learning machine with a well-defined Hamiltonian. We find that learning dynamics organize the network into a solid-liquid-solid structure, where boundary layers are constrained while central layers remain flexible. These layers exhibit distinct dynamical behaviors, including timescale separation and nonequilibrium aging. We show that the solution space is structureless in liquid layers but hierarchically organized in solid layers, indicating a direct link between dynamics and landscape geometry. These results suggest a physical perspective for generalization, arising from the interplay between flexible representations and boundary-induced constraints.