Proceedings of Machine Learning and Systems ...最新文献

筛选

英文中文

Attribution-based Sparse Activation in Large Language Models. 大型语言模型中基于归因的稀疏激活。

Proceedings of Machine Learning and Systems ... Pub Date : 2026-01-01

Jifeng Song, Xiangyu Yin, Boyuan Yang, Kai Huang, Weichen Liu, Wei Gao

{"title":"Attribution-based Sparse Activation in Large Language Models.","authors":"Jifeng Song, Xiangyu Yin, Boyuan Yang, Kai Huang, Weichen Liu, Wei Gao","doi":"","DOIUrl":"","url":null,"abstract":"LLM inference is computationally expensive due to the LLM's large parameter sizes. Existing techniques reduce the computing cost via model retraining, but cannot well adapt to different downstream tasks or variant input data at runtime. To avoid such retraining efforts for runtime adaptability, a better option is sparse activation that selectively deactivates an input-dependent set of neurons in inference, but current methods of lossless sparse activation only deactivate neurons with zero output magnitudes, and are ineffective on recent LLMs with higher parameter efficiency. In this paper, we present a new technique of attribution-based sparse activation, which is a lossy sparse activation technique that deactivates neurons with low attribution scores and aims to achieve the best tradeoff between model accuracy and computing costs. To ensure optimal sparse activation, we quantified the large errors of existing attribution metrics when used for sparse activation, due to the interdependency among attribution scores of different neurons, and further proposed a new attribution metric that can provably correct such errors. Experiments show that our technique can achieve up to 70% model sparsity in difficult generative tasks such as question answering and text summarization with <5% model accuracy loss. Such high model sparsity enables us to reduce the computing latency and memory use of LLM inference by 35% and 40%, respectively.","PeriodicalId":524198,"journal":{"name":"Proceedings of Machine Learning and Systems ...","volume":"8 ","pages":"1685-1700"},"PeriodicalIF":0.0,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13570667/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148957822","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

PLayer-FL: A Principled Approach to Personalized Layer-wise Cross-Silo Federated Learning. PLayer-FL：个性化分层跨竖井联合学习的原则方法。

Proceedings of Machine Learning and Systems ... Pub Date : 2026-01-01

Ahmed Elhussein, Gamze Gürsoy

{"title":"PLayer-FL: A Principled Approach to Personalized Layer-wise Cross-Silo Federated Learning.","authors":"Ahmed Elhussein, Gamze Gürsoy","doi":"","DOIUrl":"","url":null,"abstract":"Non-identically distributed data is a major challenge in Federated Learning (FL). Personalized FL tackles this by balancing local model adaptation with global model consistency. One variant, partial FL, leverages the observation that early layers learn more transferable features by federating only early layers. However, current partial FL approaches use predetermined, architecture-specific rules to select layers, limiting their applicability. We introduce Principled Layer-wise-FL (PLayer-FL), which uses a novel federation sensitivity metric to identify layers that benefit from federation. This metric, inspired by model pruning, quantifies each layer's contribution to cross-client generalization after the first training epoch, identifying a transition point in the network where the benefits of federation diminish. We first demonstrate that our federation sensitivity metric shows strong correlation with established generalization measures across diverse architectures. Next, we show that PLayer-FL outperforms existing FL algorithms on a range of tasks, also achieving more uniform performance improvements across clients.","PeriodicalId":524198,"journal":{"name":"Proceedings of Machine Learning and Systems ...","volume":"8 ","pages":"1206-1217"},"PeriodicalIF":0.0,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13570376/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148957823","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0