A comprehensive evaluation framework for synthetic medical tabular data generation

Abstract

Machine learning (ML) applications have enabled significant advancements in healthcare, such as predicting pandemics, personalizing treatments, and developing life-saving drugs. However, ML model training requires large datasets, which are difficult to obtain in healthcare due to privacy concerns. Synthetic data generation offers a promising solution by providing access to large-scale training data while protecting patient privacy. Our research focuses on tabular medical data, the predominant format for Electronic Health Records (EHRs), and introduces a comprehensive evaluation framework that assesses synthetic data in four critical dimensions: quality, privacy, usability, and computational complexity of the data generation process. The framework ensures that synthetic data maintains sufficient similarity to real data for ML applications while preserving patient confidentiality. To validate our approach, we applied six state-of-the-art (SOTA) generative models to generate synthetic medical datasets and evaluated them within our framework. In contrast to conventional approaches that focus primarily on statistical similarity, our framework provides a broader assessment that incorporates outlier detection, privacy risks, and domain-specific constraints. Our findings demonstrate that our framework can identify critical shortcomings in synthetic data generation models, such as the amplification of duplicate rows and the generation of out-of-range values, which are overlooked by traditional statistical evaluation methods. Our implementation of the framework is available at: https://github.com/akurakova/SDE_Framework