An efficient and distribution-free symmetry test for high-dimensional data based on energy statistics and random projections

IF 1.6 3区 数学 Q3 COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS
Bo Chen , Feifei Chen , Junxin Wang , Tao Qiu
{"title":"An efficient and distribution-free symmetry test for high-dimensional data based on energy statistics and random projections","authors":"Bo Chen ,&nbsp;Feifei Chen ,&nbsp;Junxin Wang ,&nbsp;Tao Qiu","doi":"10.1016/j.csda.2024.108123","DOIUrl":null,"url":null,"abstract":"<div><div>Testing the departures from symmetry is a critical issue in statistics. Over the last two decades, substantial effort has been invested in developing tests for central symmetry in multivariate and high-dimensional contexts. Traditional tests, which rely on Euclidean distance, face significant challenges in high-dimensional data. These tests struggle to capture overall central symmetry and are often limited to verifying whether the distribution's center aligns with the coordinate origin, a problem exacerbated by the “curse of dimensionality.” Furthermore, they tend to be computationally intensive, often making them impractical for large datasets. To overcome these limitations, we propose a nonparametric test based on the random projected energy distance, extending the energy distance test through random projections. This method effectively reduces data dimensions by projecting high-dimensional data onto lower-dimensional spaces, with the randomness ensuring maximum preservation of information. Theoretically, as the number of random projections approaches infinity, the risk of power loss from inadequate directions is mitigated. Leveraging <em>U</em>-statistic theory, our test's asymptotic null distribution is standard normal, which holds true regardless of the data dimensionality relative to sample size, thus eliminating the need for re-sampling to determine critical values. For computational efficiency with large datasets, we adopt a divide-and-average strategy, partitioning the data into smaller blocks of size <em>m</em>. Within each block, the estimates of the random projected energy distance are normally distributed. By averaging these estimates across all blocks, we derive a test statistic that is asymptotically standard normal. This method significantly reduces computational complexity from quadratic to linear in sample size, enhancing the feasibility of our test for extensive data analysis. Through extensive numerical studies, we have demonstrated the robust empirical performance of our test in terms of size and power, affirming its practical utility in statistical applications for high-dimensional data.</div></div>","PeriodicalId":55225,"journal":{"name":"Computational Statistics & Data Analysis","volume":"206 ","pages":"Article 108123"},"PeriodicalIF":1.6000,"publicationDate":"2025-01-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Computational Statistics & Data Analysis","FirstCategoryId":"100","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S016794732400207X","RegionNum":3,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS","Score":null,"Total":0}
引用次数: 0

Abstract

Testing the departures from symmetry is a critical issue in statistics. Over the last two decades, substantial effort has been invested in developing tests for central symmetry in multivariate and high-dimensional contexts. Traditional tests, which rely on Euclidean distance, face significant challenges in high-dimensional data. These tests struggle to capture overall central symmetry and are often limited to verifying whether the distribution's center aligns with the coordinate origin, a problem exacerbated by the “curse of dimensionality.” Furthermore, they tend to be computationally intensive, often making them impractical for large datasets. To overcome these limitations, we propose a nonparametric test based on the random projected energy distance, extending the energy distance test through random projections. This method effectively reduces data dimensions by projecting high-dimensional data onto lower-dimensional spaces, with the randomness ensuring maximum preservation of information. Theoretically, as the number of random projections approaches infinity, the risk of power loss from inadequate directions is mitigated. Leveraging U-statistic theory, our test's asymptotic null distribution is standard normal, which holds true regardless of the data dimensionality relative to sample size, thus eliminating the need for re-sampling to determine critical values. For computational efficiency with large datasets, we adopt a divide-and-average strategy, partitioning the data into smaller blocks of size m. Within each block, the estimates of the random projected energy distance are normally distributed. By averaging these estimates across all blocks, we derive a test statistic that is asymptotically standard normal. This method significantly reduces computational complexity from quadratic to linear in sample size, enhancing the feasibility of our test for extensive data analysis. Through extensive numerical studies, we have demonstrated the robust empirical performance of our test in terms of size and power, affirming its practical utility in statistical applications for high-dimensional data.
基于能量统计和随机投影的高维数据的一种有效且无分布的对称性检验
检验偏离对称性是统计学中的一个关键问题。在过去的二十年里,人们投入了大量的精力来开发多元和高维环境下的中心对称测试。传统的基于欧几里得距离的测试在高维数据中面临重大挑战。这些测试很难捕捉到整体的中心对称性,并且通常仅限于验证分布的中心是否与坐标原点对齐,“维度诅咒”加剧了这个问题。此外,它们往往是计算密集型的,对于大型数据集来说往往是不切实际的。为了克服这些局限性,我们提出了一种基于随机投影能量距离的非参数检验方法,将随机投影的能量距离检验方法进行了扩展。该方法通过将高维数据投影到低维空间,有效地降低了数据维数,随机性保证了信息的最大保留。理论上,当随机投影的数量接近无穷大时,不适当方向造成的功率损失的风险就会降低。利用u统计理论,我们的检验的渐近零分布是标准正态分布,无论相对于样本量的数据维度如何,它都成立,从而消除了重新抽样以确定临界值的需要。为了提高大型数据集的计算效率,我们采用了除平均策略,将数据划分为大小为m的较小块。在每个块内,随机投影能量距离的估计值是正态分布的。通过在所有块上平均这些估计值,我们得到一个渐近标准正态的检验统计量。该方法显著降低了样本大小从二次型到线性型的计算复杂度,增强了我们测试广泛数据分析的可行性。通过广泛的数值研究,我们已经证明了我们的测试在大小和功率方面的强大的经验性能,肯定了它在高维数据统计应用中的实际效用。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
Computational Statistics & Data Analysis
Computational Statistics & Data Analysis 数学-计算机:跨学科应用
CiteScore
3.70
自引率
5.60%
发文量
167
审稿时长
60 days
期刊介绍: Computational Statistics and Data Analysis (CSDA), an Official Publication of the network Computational and Methodological Statistics (CMStatistics) and of the International Association for Statistical Computing (IASC), is an international journal dedicated to the dissemination of methodological research and applications in the areas of computational statistics and data analysis. The journal consists of four refereed sections which are divided into the following subject areas: I) Computational Statistics - Manuscripts dealing with: 1) the explicit impact of computers on statistical methodology (e.g., Bayesian computing, bioinformatics,computer graphics, computer intensive inferential methods, data exploration, data mining, expert systems, heuristics, knowledge based systems, machine learning, neural networks, numerical and optimization methods, parallel computing, statistical databases, statistical systems), and 2) the development, evaluation and validation of statistical software and algorithms. Software and algorithms can be submitted with manuscripts and will be stored together with the online article. II) Statistical Methodology for Data Analysis - Manuscripts dealing with novel and original data analytical strategies and methodologies applied in biostatistics (design and analytic methods for clinical trials, epidemiological studies, statistical genetics, or genetic/environmental interactions), chemometrics, classification, data exploration, density estimation, design of experiments, environmetrics, education, image analysis, marketing, model free data exploration, pattern recognition, psychometrics, statistical physics, image processing, robust procedures. [...] III) Special Applications - [...] IV) Annals of Statistical Data Science [...]
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信