The collection of relevant information about the vulnerability of infrastructure damaged by landslides is not an easy task due to the existence of several compounding factors and uncertainties. This makes it difficult to quantitatively estimate their vulnerability to slow-moving landslides. This paper presents a new vulnerability assessment model for masonry buildings on slow-moving landslides based on physical models and field observations. A masonry building model is made of brick and concrete at a scale of 1:10 to physically simulate the damage in structures caused by ground tension cracks commonly developed on slow-moving landslides. The tension crack opening process is simulated through a load-controlled table with an aperture on which the building model is constructed. The strain on the wall and its foundation were measured, and the damage of the model (crack formation and evolution for each loading step) was collected, described, and analyzed. These data were used to develop failure criteria for masonry buildings in rural areas in China in terms of a quantitative vulnerability curve. The quantitative model of vulnerability for masonry structures was established based on fuzzy mathematics and the Weibull function applied on the test data and observations. The vulnerability curve is verified with field cases of masonry buildings damaged by ground tension cracks associated with slow-moving landslides in the Three Gorges Reservoir area. The results support further testing and use of vulnerability curve proposed.