meMIA: Multilevel Ensemble Membership Inference Attack

Abstract

Leakage of private information in machine learning models can lead to breaches of confidentiality, identity theft, and unauthorized access to personal data. Ensuring the safe and trustworthy deployment of AI systems necessitates addressing privacy concerns to prevent unintentional disclosure and discrimination. One significant threat, membership inference (MI) attacks, exploit vulnerabilities in target learning models to determine if a given sample was part of the training set. However, the effectiveness of existing MI attacks is often limited by the number of classes in the dataset or the need for diverse multilevel adversarial features to exploit overfitted models. To enhance MI attack performance, we propose meMIA, a novel framework based on stacked ensemble learning. meMIA integrates embeddings from a neural network (NN) and a long short-term memory (LSTM) model, training a subsequent NN, termed the meta-model, on the concatenated embeddings. This method leverages the complementary strengths of NN and LSTM models; the LSTM captures order differences in confidence scores, while the NN discerns probability distribution differences between member and nonmember samples. We extensively evaluate meMIA on seven benchmark datasets, demonstrating that it surpasses current state-of-the-art MI attacks, achieving accuracy up to 94.6% and near-perfect recall. meMIA's superior performance, especially on datasets with fewer classes, underscores the urgent need for robust defenses against privacy attacks in machine learning, contributing to the safer and more ethical use of AI technologies.