The interface between a track slab (TS) and self-compacting concrete (SCC) is the weakest zone of the China Railway Track System (CRTS) III slab track. One of the most promising approaches for reducing interfacial fatigue damage under the long-term repetitive effects of environmental and train loads is the incorporation of organic polymer latex into SCC. In this study, the fatigue behavior and damage modeling of the track slab-SCC interface (TS-SCC0) and track slab-polymer-modified SCC interface (TS-SCC20) were investigated through three-point bending tests. The results demonstrated that most of the TS-SCC0 and TS-SCC20 interfaces failed in the mode of mortar failure, that is, a thin layer of mortar attaching to the TS side. The proportion of aggregate-mortar debonding failure reduced from 4.09% to 1.86% upon the incorporation of polymer latex, suggesting that the polymer latex contributed to the enhancement of the bonding strength within the mortar-aggregate transition zone. The average fatigue life of TS-SCC20 was 2.27–3.52 times longer than that of TS-SCC0 under the same stress level. The evolution of interface strain was divided into three stages: rapid growth, stable growth, and instability. The proportion of the third stage in TS-SCC20 was 1.67 times higher than that in TS-SCC0, indicating the hindrance of polymer to crack propagation. Moreover, a generalized interfacial fatigue damage model linking the damage index to the fatigue life ratio at various stress levels was established to quantitatively study the fatigue damage process of the TS-SCC20 interface and to serve as an interfacial fatigue constitutive model in finite element analysis.