超越 LoRA:探索时间序列基础模型的高效微调技术

Divij Gupta, Anubhav Bhatti, Surajsinh Parmar
{"title":"超越 LoRA:探索时间序列基础模型的高效微调技术","authors":"Divij Gupta, Anubhav Bhatti, Surajsinh Parmar","doi":"arxiv-2409.11302","DOIUrl":null,"url":null,"abstract":"Time Series Foundation Models (TSFMs) have recently garnered attention for\ntheir ability to model complex, large-scale time series data across domains\nsuch as retail, finance, and transportation. However, their application to\nsensitive, domain-specific fields like healthcare remains challenging,\nprimarily due to the difficulty of fine-tuning these models for specialized,\nout-of-domain tasks with scarce publicly available datasets. In this work, we\nexplore the use of Parameter-Efficient Fine-Tuning (PEFT) techniques to address\nthese limitations, focusing on healthcare applications, particularly ICU vitals\nforecasting for sepsis patients. We introduce and evaluate two selective\n(BitFit and LayerNorm Tuning) and two additive (VeRA and FourierFT) PEFT\ntechniques on multiple configurations of the Chronos TSFM for forecasting vital\nsigns of sepsis patients. Our comparative analysis demonstrates that some of\nthese PEFT methods outperform LoRA in terms of parameter efficiency and domain\nadaptation, establishing state-of-the-art (SOTA) results in ICU vital\nforecasting tasks. Interestingly, FourierFT applied to the Chronos (Tiny)\nvariant surpasses the SOTA model while fine-tuning only 2,400 parameters\ncompared to the 700K parameters of the benchmark.","PeriodicalId":501301,"journal":{"name":"arXiv - CS - Machine Learning","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2024-09-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Beyond LoRA: Exploring Efficient Fine-Tuning Techniques for Time Series Foundational Models\",\"authors\":\"Divij Gupta, Anubhav Bhatti, Surajsinh Parmar\",\"doi\":\"arxiv-2409.11302\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Time Series Foundation Models (TSFMs) have recently garnered attention for\\ntheir ability to model complex, large-scale time series data across domains\\nsuch as retail, finance, and transportation. However, their application to\\nsensitive, domain-specific fields like healthcare remains challenging,\\nprimarily due to the difficulty of fine-tuning these models for specialized,\\nout-of-domain tasks with scarce publicly available datasets. In this work, we\\nexplore the use of Parameter-Efficient Fine-Tuning (PEFT) techniques to address\\nthese limitations, focusing on healthcare applications, particularly ICU vitals\\nforecasting for sepsis patients. We introduce and evaluate two selective\\n(BitFit and LayerNorm Tuning) and two additive (VeRA and FourierFT) PEFT\\ntechniques on multiple configurations of the Chronos TSFM for forecasting vital\\nsigns of sepsis patients. Our comparative analysis demonstrates that some of\\nthese PEFT methods outperform LoRA in terms of parameter efficiency and domain\\nadaptation, establishing state-of-the-art (SOTA) results in ICU vital\\nforecasting tasks. Interestingly, FourierFT applied to the Chronos (Tiny)\\nvariant surpasses the SOTA model while fine-tuning only 2,400 parameters\\ncompared to the 700K parameters of the benchmark.\",\"PeriodicalId\":501301,\"journal\":{\"name\":\"arXiv - CS - Machine Learning\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2024-09-17\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"arXiv - CS - Machine Learning\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/arxiv-2409.11302\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - CS - Machine Learning","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.11302","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

摘要

时间序列基础模型(TSFMs)最近因其能够对零售、金融和交通等领域的复杂、大规模时间序列数据进行建模而备受关注。然而,它们在医疗保健等敏感、特定领域的应用仍然具有挑战性,这主要是由于很难针对专业、领域外的任务以及稀缺的公开可用数据集对这些模型进行微调。在这项工作中,我们探讨了如何利用参数高效微调(PEFT)技术来解决这些局限性,重点是医疗保健应用,尤其是重症监护室脓毒症患者的生命预报。我们在 Chronos TSFM 的多种配置上引入并评估了两种选择性(BitFit 和 LayerNorm Tuning)和两种添加性(VeRA 和 FourierFT)PEFT 技术,用于预测败血症患者的生命体征。我们的比较分析表明,其中一些 PEFT 方法在参数效率和领域适应性方面优于 LoRA,在 ICU 生命体征预测任务中取得了最先进(SOTA)的结果。有趣的是,应用于 Chronos(Tiny)变体的 FourierFT 仅微调了 2,400 个参数,就超越了 SOTA 模型,而基准模型需要微调 70 万个参数。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
Beyond LoRA: Exploring Efficient Fine-Tuning Techniques for Time Series Foundational Models
Time Series Foundation Models (TSFMs) have recently garnered attention for their ability to model complex, large-scale time series data across domains such as retail, finance, and transportation. However, their application to sensitive, domain-specific fields like healthcare remains challenging, primarily due to the difficulty of fine-tuning these models for specialized, out-of-domain tasks with scarce publicly available datasets. In this work, we explore the use of Parameter-Efficient Fine-Tuning (PEFT) techniques to address these limitations, focusing on healthcare applications, particularly ICU vitals forecasting for sepsis patients. We introduce and evaluate two selective (BitFit and LayerNorm Tuning) and two additive (VeRA and FourierFT) PEFT techniques on multiple configurations of the Chronos TSFM for forecasting vital signs of sepsis patients. Our comparative analysis demonstrates that some of these PEFT methods outperform LoRA in terms of parameter efficiency and domain adaptation, establishing state-of-the-art (SOTA) results in ICU vital forecasting tasks. Interestingly, FourierFT applied to the Chronos (Tiny) variant surpasses the SOTA model while fine-tuning only 2,400 parameters compared to the 700K parameters of the benchmark.
求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信