With the improvements in our knowledge of the phytolith morphology of different Poaceae subfamilies and even different species, phytolith records in paleosols provide valuable information on past vegetation in situ, especially for herbaceous plants. Nevertheless, the reliability of phytolith records for the reconstruction of forest vegetation needs to be further verified, and it needs to be assessed before they can be used in palaeoecological contexts. In this study we analyzed the topsoil phytolith assemblages of 39 sampling sites and established a modern soil phytolith reference dataset for five plant communities in the mid-subtropical zone of China. We then used the phytolith reference dataset to analyze a stratigraphic sequence of fossil phytolith assemblages from the same region. The results show that the topsoil phytolith assemblages and phytolith indices (Ic and Iph) vary substantially among different plant communities, and that there is a good correlation with the aboveground vegetation composition. This shows that modern soil phytoliths are reliable analogues for reconstructing the paleovegetation. The results of a paleovegetation reconstruction based on the modern phytolith reference dataset show that the SWGT region has experienced substantial vegetation changes during the Holocene—from evergreen and deciduous broadleaved mixed forest to evergreen broadleaved forest, and finally to evergreen broadleaved forest with occasional shrub and herbaceous communities. The phytolith-based Holocene vegetation types are consistent with those reconstructed by a pollen record from the same stratigraphic profile. Moreover, the phytolith reconstruction also records changes in the understory herbaceous plants and C3/C4 grasses, which supplements the results of the pollen-based paleovegetation reconstruction. Overall, our results confirm the reliability of phytolith analysis for interpreting the entire vegetation landscape and forest composition in the mid-subtropical zone of China. They also provide a new perspective for the application of phytoliths to the study of forest dynamics in this region.