This paper establishes a variable order fractional creep model for asphalt mixtures based on rheology and fractional calculus theory and demonstrates the characterization of variable order fractional dashpot's memory effect and creep response on viscoelastic materials. Uniaxial compression creep tests at different stresses are carried out, and the variable viscoelastic mechanical properties and damage evolution of asphalt mixtures are analyzed and interpreted at a mesoscopic level. The order of the variable order fractional creep model is calculated, and the physical significance of the order and the properties of the order function are investigated in conjunction with the damage development curve. The results show that the derived variable order fractional creep model has a simple form and can accurately describe the complex three-stage creep response of asphalt mixtures. During the creep process, the model order always tends to decrease and then increase, which is similar to the development law of asphalt mixture damage, and it is speculated that the order can characterize the evolution of viscoelastic mechanical properties and damage. Normalization shows that the order curves of the three-stage creep at different stresses almost overlap and can be fitted by the quadratic function.