Gradient-Free De Novo Learning.

Abstract

This technical note applies active inference to the problem of learning goal-directed behaviour from scratch, namely, de novo learning. By de novo learning, we mean discovering, directly from observations, the structure and parameters of a discrete generative model for sequential policy optimisation. Concretely, our procedure grows and then reduces a model until it discovers a pullback attractor over (generalised) states; this attracting set supplies paths of least action among goal states while avoiding costly states. The implicit efficiency rests upon reframing the learning problem through the lens of the free energy principle, under which it is sufficient to learn a generative model whose dynamics feature such an attracting set. For context, we briefly relate this perspective to value-based formulations (e.g., Bellman optimality) and then apply the active inference formulation to a small arcade game to illustrate de novo structure learning and ensuing agency.