Proceedings of the First Workshop on Federated Learning for Natural Language Processing (FL4NLP 2022)最新文献

筛选
英文 中文
ActPerFL: Active Personalized Federated Learning ActPerFL:主动个性化联邦学习
Huili Chen, Jie Ding, Eric W. Tramel, Shuang Wu, Anit Kumar Sahu, S. Avestimehr, Tao Zhang
{"title":"ActPerFL: Active Personalized Federated Learning","authors":"Huili Chen, Jie Ding, Eric W. Tramel, Shuang Wu, Anit Kumar Sahu, S. Avestimehr, Tao Zhang","doi":"10.18653/v1/2022.fl4nlp-1.1","DOIUrl":"https://doi.org/10.18653/v1/2022.fl4nlp-1.1","url":null,"abstract":"In the context of personalized federated learning (FL), the critical challenge is to balance local model improvement and global model tuning when the personal and global objectives may not be exactly aligned. Inspired by Bayesian hierarchical models, we develop ActPerFL, a self-aware personalized FL method where each client can automatically balance the training of its local personal model and the global model that implicitly contributes to other clients’ training. Such a balance is derived from the inter-client and intra-client uncertainty quantification. Consequently, ActPerFL can adapt to the underlying clients’ heterogeneity with uncertainty-driven local training and model aggregation. With experimental studies on Sent140 and Amazon Alexa audio data, we show that ActPerFL can achieve superior personalization performance compared with the existing counterparts.","PeriodicalId":438236,"journal":{"name":"Proceedings of the First Workshop on Federated Learning for Natural Language Processing (FL4NLP 2022)","volume":"90 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131866610","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 8
Adaptive Differential Privacy for Language Model Training 语言模型训练的自适应差分隐私
Xinwei Wu, Li Gong, Deyi Xiong
{"title":"Adaptive Differential Privacy for Language Model Training","authors":"Xinwei Wu, Li Gong, Deyi Xiong","doi":"10.18653/v1/2022.fl4nlp-1.3","DOIUrl":"https://doi.org/10.18653/v1/2022.fl4nlp-1.3","url":null,"abstract":"Although differential privacy (DP) can protect language models from leaking privacy, its indiscriminative protection on all data points reduces its practical utility. Previous works improve DP training by discriminating privacy and non-privacy data. But these works rely on datasets with prior privacy information, which is not available in real-world scenarios. In this paper, we propose an Adaptive Differential Privacy (ADP) framework for language modeling without resorting to prior privacy information. We estimate the probability that a linguistic item contains privacy based on a language model. We further propose a new Adam algorithm that adjusts the degree of differential privacy noise injected to the language model according to the estimated privacy probabilities. Experiments demonstrate that our ADP improves differentially private language modeling to achieve good protection from canary attackers.","PeriodicalId":438236,"journal":{"name":"Proceedings of the First Workshop on Federated Learning for Natural Language Processing (FL4NLP 2022)","volume":"72 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127269371","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 1
Intrinsic Gradient Compression for Scalable and Efficient Federated Learning 面向可扩展高效联邦学习的固有梯度压缩
Luke Melas-Kyriazi, Franklyn Wang
{"title":"Intrinsic Gradient Compression for Scalable and Efficient Federated Learning","authors":"Luke Melas-Kyriazi, Franklyn Wang","doi":"10.18653/v1/2022.fl4nlp-1.4","DOIUrl":"https://doi.org/10.18653/v1/2022.fl4nlp-1.4","url":null,"abstract":"Federated learning is a rapidly growing area of research, holding the promise of privacy-preserving distributed training on edge devices. The largest barrier to wider adoption of federated learning is the communication cost of model updates, which is accentuated by the fact that many edge devices are bandwidth-constrained. At the same time, within the machine learning theory community, a separate line of research has emerged around optimizing networks within a subspace of the full space of all parameters. The dimension of the smallest subspace for which these methods still yield strong results is called the intrinsic dimension. In this work, we prove a general correspondence between the notions of intrinsic dimension and gradient compressibility, and we show that a family of low-bandwidth federated learning algorithms, which we call intrinsic gradient compression algorithms, naturally emerges from this correspondence. Finally, we conduct large-scale NLP experiments using transformer models with over 100M parameters (GPT-2 and BERT), and show that our method significantly outperforms the state-of-the-art in gradient compression.","PeriodicalId":438236,"journal":{"name":"Proceedings of the First Workshop on Federated Learning for Natural Language Processing (FL4NLP 2022)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131365491","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 4
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
相关产品
×
本文献相关产品
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信
小红书