从少量样本中进行高维协方差估计

IF 4.4 2区 地球科学 Q1 METEOROLOGY & ATMOSPHERIC SCIENCES
David Vishny, Matthias Morzfeld, Kyle Gwirtz, Eviatar Bach, Oliver R. A. Dunbar, Daniel Hodyss
{"title":"从少量样本中进行高维协方差估计","authors":"David Vishny,&nbsp;Matthias Morzfeld,&nbsp;Kyle Gwirtz,&nbsp;Eviatar Bach,&nbsp;Oliver R. A. Dunbar,&nbsp;Daniel Hodyss","doi":"10.1029/2024MS004417","DOIUrl":null,"url":null,"abstract":"<p>We synthesize knowledge from numerical weather prediction, inverse theory, and statistics to address the problem of estimating a high-dimensional covariance matrix from a small number of samples. This problem is fundamental in statistics, machine learning/artificial intelligence, and in modern Earth science. We create several new adaptive methods for high-dimensional covariance estimation, but one method, which we call Noise-Informed Covariance Estimation (NICE), stands out because it has three important properties: (a) NICE is conceptually simple and computationally efficient; (b) NICE guarantees symmetric positive semi-definite covariance estimates; and (c) NICE is largely tuning-free. We illustrate the use of NICE on a large set of Earth science–inspired numerical examples, including cycling data assimilation, inversion of geophysical field data, and training of feed-forward neural networks with time-averaged data from a chaotic dynamical system. Our theory, heuristics and numerical tests suggest that NICE may indeed be a viable option for high-dimensional covariance estimation in many Earth science problems.</p>","PeriodicalId":14881,"journal":{"name":"Journal of Advances in Modeling Earth Systems","volume":"16 9","pages":""},"PeriodicalIF":4.4000,"publicationDate":"2024-08-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1029/2024MS004417","citationCount":"0","resultStr":"{\"title\":\"High-Dimensional Covariance Estimation From a Small Number of Samples\",\"authors\":\"David Vishny,&nbsp;Matthias Morzfeld,&nbsp;Kyle Gwirtz,&nbsp;Eviatar Bach,&nbsp;Oliver R. A. Dunbar,&nbsp;Daniel Hodyss\",\"doi\":\"10.1029/2024MS004417\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<p>We synthesize knowledge from numerical weather prediction, inverse theory, and statistics to address the problem of estimating a high-dimensional covariance matrix from a small number of samples. This problem is fundamental in statistics, machine learning/artificial intelligence, and in modern Earth science. We create several new adaptive methods for high-dimensional covariance estimation, but one method, which we call Noise-Informed Covariance Estimation (NICE), stands out because it has three important properties: (a) NICE is conceptually simple and computationally efficient; (b) NICE guarantees symmetric positive semi-definite covariance estimates; and (c) NICE is largely tuning-free. We illustrate the use of NICE on a large set of Earth science–inspired numerical examples, including cycling data assimilation, inversion of geophysical field data, and training of feed-forward neural networks with time-averaged data from a chaotic dynamical system. Our theory, heuristics and numerical tests suggest that NICE may indeed be a viable option for high-dimensional covariance estimation in many Earth science problems.</p>\",\"PeriodicalId\":14881,\"journal\":{\"name\":\"Journal of Advances in Modeling Earth Systems\",\"volume\":\"16 9\",\"pages\":\"\"},\"PeriodicalIF\":4.4000,\"publicationDate\":\"2024-08-30\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"https://onlinelibrary.wiley.com/doi/epdf/10.1029/2024MS004417\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Journal of Advances in Modeling Earth Systems\",\"FirstCategoryId\":\"89\",\"ListUrlMain\":\"https://onlinelibrary.wiley.com/doi/10.1029/2024MS004417\",\"RegionNum\":2,\"RegionCategory\":\"地球科学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"METEOROLOGY & ATMOSPHERIC SCIENCES\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Advances in Modeling Earth Systems","FirstCategoryId":"89","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1029/2024MS004417","RegionNum":2,"RegionCategory":"地球科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"METEOROLOGY & ATMOSPHERIC SCIENCES","Score":null,"Total":0}
引用次数: 0

摘要

我们综合了数值天气预报、逆理论和统计学的知识,以解决从少量样本中估计高维协方差矩阵的问题。这个问题是统计学、机器学习/人工智能和现代地球科学的基础。我们为高维协方差估计创建了几种新的自适应方法,但其中一种我们称为噪声信息协方差估计(NICE)的方法脱颖而出,因为它具有三个重要特性:(a)NICE 概念简单,计算效率高;(b)NICE 保证对称正半有限协方差估计;(c)NICE 基本上无需调整。我们在大量受地球科学启发的数值示例中说明了 NICE 的使用,包括循环数据同化、地球物理场数据反演以及利用混沌动力系统的时均数据训练前馈神经网络。我们的理论、启发式方法和数值测试表明,NICE 可能确实是许多地球科学问题中进行高维协方差估计的可行选择。
本文章由计算机程序翻译,如有差异,请以英文原文为准。

High-Dimensional Covariance Estimation From a Small Number of Samples

High-Dimensional Covariance Estimation From a Small Number of Samples

We synthesize knowledge from numerical weather prediction, inverse theory, and statistics to address the problem of estimating a high-dimensional covariance matrix from a small number of samples. This problem is fundamental in statistics, machine learning/artificial intelligence, and in modern Earth science. We create several new adaptive methods for high-dimensional covariance estimation, but one method, which we call Noise-Informed Covariance Estimation (NICE), stands out because it has three important properties: (a) NICE is conceptually simple and computationally efficient; (b) NICE guarantees symmetric positive semi-definite covariance estimates; and (c) NICE is largely tuning-free. We illustrate the use of NICE on a large set of Earth science–inspired numerical examples, including cycling data assimilation, inversion of geophysical field data, and training of feed-forward neural networks with time-averaged data from a chaotic dynamical system. Our theory, heuristics and numerical tests suggest that NICE may indeed be a viable option for high-dimensional covariance estimation in many Earth science problems.

求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
Journal of Advances in Modeling Earth Systems
Journal of Advances in Modeling Earth Systems METEOROLOGY & ATMOSPHERIC SCIENCES-
CiteScore
11.40
自引率
11.80%
发文量
241
审稿时长
>12 weeks
期刊介绍: The Journal of Advances in Modeling Earth Systems (JAMES) is committed to advancing the science of Earth systems modeling by offering high-quality scientific research through online availability and open access licensing. JAMES invites authors and readers from the international Earth systems modeling community. Open access. Articles are available free of charge for everyone with Internet access to view and download. Formal peer review. Supplemental material, such as code samples, images, and visualizations, is published at no additional charge. No additional charge for color figures. Modest page charges to cover production costs. Articles published in high-quality full text PDF, HTML, and XML. Internal and external reference linking, DOI registration, and forward linking via CrossRef.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信