The key challenge of graph-based multi-view clustering methods lies in how to capture a consensus clustering structure. Although existing methods have achieved good performances, they still share the following limitations: 1) The high computational complexity caused by large graph leaning. 2) The contaminated information in different views reduces the consistency of the fused graph. 3) The two-stage clustering strategy leads to sub-optimal solutions and error accumulation. To solve the above issues, we propose a novel multi-view clustering algorithm termed Multi-View Clustering with Filtered Bipartite Graph (MVC-FBG). In the graph construction stage, we select representative anchors to construct anchor graphs with less space complexity. Then we explicitly filter out the contaminated information to preserve the consistency in different views. Moreover, a low-rank constraint is imposed on the Laplacian matrix of the unified graph to obtain the clustering results directly. Furthermore, we design an efficient alternating optimization algorithm to solve our model, which enjoys a linear time complexity that can scale well with the data size. Extensive experimental results on different scale datasets demonstrate the effectiveness and efficiency of our proposed method.