Large-Scale Data Quality Challenges, Framework and Evaluation in Metro Systems

Abstract

Data quality is a fundamental challenge for downstream data mining tasks. While numerous studies have addressed data quality issues in various contexts, there is a notable lack of systematic research on data quality in metro systems. Metro systems generate a vast volume of multisource heterogeneous datasets daily, and many data mining tasks have been developed for operational and management purposes. Therefore, investigating data quality problems in metro systems is crucial. In this paper, we systematically explore data quality issues in metro systems. First, we present a comprehensive analysis method to examine data quality problems such as missing data, noise, and weak semantics. Second, we design five metrics to measure data quality and propose a set of quality improvement approaches. These approaches include a travel pattern-based missing value imputation method, a heuristic trajectory noise filtering method, and a data semantics enhancement method. Additionally, we develop an automated pipeline solution where the data quality enhancement algorithms are seamlessly integrated with the data processing pipeline. Finally, we provide a case study to illustrate the significant benefits of our data quality improvement methods. We conducted extensive experiments to validate our methods on a set of large-scale datasets collected from a metro system, which include, Wi-Fi signal data, and electronic fence data. The results indicate that 1) the proposed imputation method surpasses other baselines by 26.47% to 44.82%; 2) the proposed noise filtering method outperforms other baselines by an average of 12.22%; and 3) the proposed data semantics enrichment method exceeds the baseline method by 37.34% in terms of maximum accuracy.