IEEE Journal of Selected Topics in Signal Processing最新文献_第9页

Speech Acoustic Markers Can Detect Mild Cognitive Impairment in Parkinson’s Disease 语音声学标记可以检测帕金森病的轻度认知障碍

IF 13.7 1区工程技术

IEEE Journal of Selected Topics in Signal Processing Pub Date : 2025-10-17 DOI: 10.1109/JSTSP.2025.3620716

Kara M. Smith;James R. Williamson;Thomas F. Quatieri

{"title":"Speech Acoustic Markers Can Detect Mild Cognitive Impairment in Parkinson’s Disease","authors":"Kara M. Smith;James R. Williamson;Thomas F. Quatieri","doi":"10.1109/JSTSP.2025.3620716","DOIUrl":"https://doi.org/10.1109/JSTSP.2025.3620716","url":null,"abstract":"Background: Speech biomarkers have been used to assess motor dysfunction in people with Parkinson’s disease (PD), but speech biomarkers for mild cognitive impairment in PD (PD-MCI) have not been well studied. Objective: To identify speech acoustic features associated with PD-MCI and evaluate the performance of a model to discriminate PD-MCI from participants with normal cognitive status (PD-NC). Methods: We analyzed speech samples from 42 participants with PD, diagnosed as either PD-MCI or PD-NC using the Movement disorders Society Task Force Tier II criteria as a gold-standard classification of MCI. A reading passage and a picture description task were analyzed for acoustic features, which were used to generate individual and then a final fused Gaussian mixture model (GMM) to discriminate PD-MCI and PD-NC participants. Results: The picture description task yielded a larger number of acoustic features that were highly associated with PD-MCI status compared to the reading task. Fusing the model outputs from the picture description task resulted in an AUC = 0.82 for discriminating PD-MCI from PD-NC participants. The acoustic features associated with PD-MCI stemmed from multiple speech subsystems. Conclusion: PD-MCI has a distinct speech acoustic signature that may be harnessed to develop better tools to detect and monitor this complication.","PeriodicalId":13038,"journal":{"name":"IEEE Journal of Selected Topics in Signal Processing","volume":"19 5","pages":"731-740"},"PeriodicalIF":13.7,"publicationDate":"2025-10-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=11206405","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145659224","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Detecting Neurocognitive Disorders Through Analyses of Topic Evolution and Cross-Modal Consistency in Visual-Stimulated Narratives 通过分析视觉刺激叙事的主题演变和跨模态一致性来检测神经认知障碍

IF 13.7 1区工程技术

IEEE Journal of Selected Topics in Signal Processing Pub Date : 2025-10-15 DOI: 10.1109/JSTSP.2025.3622049

Jinchao Li;Yuejiao Wang;Junan Li;Jiawen Kang;Bo Zheng;Ka Ho Wong;Brian Kan-Wing Mak;Helene H. Fung;Jean Woo;Man-Wai Mak;Timothy Kwok;Vincent Mok;Xianmin Gong;Xixin Wu;Xunying Liu;Patrick C. M. Wong;Helen Meng

{"title":"Detecting Neurocognitive Disorders Through Analyses of Topic Evolution and Cross-Modal Consistency in Visual-Stimulated Narratives","authors":"Jinchao Li;Yuejiao Wang;Junan Li;Jiawen Kang;Bo Zheng;Ka Ho Wong;Brian Kan-Wing Mak;Helene H. Fung;Jean Woo;Man-Wai Mak;Timothy Kwok;Vincent Mok;Xianmin Gong;Xixin Wu;Xunying Liu;Patrick C. M. Wong;Helen Meng","doi":"10.1109/JSTSP.2025.3622049","DOIUrl":"https://doi.org/10.1109/JSTSP.2025.3622049","url":null,"abstract":"Early detection of neurocognitive disorders (NCDs) is crucial for timely intervention and disease management. Given that language impairments manifest early in NCD progression, visual-stimulated narrative (VSN)-based analysis offers a promising avenue for NCD detection. Current VSN-based NCD detection methods primarily focus on linguistic microstructures (e.g., lexical diversity) that are closely tied to bottom-up, stimulus-driven cognitive processes. While these features illuminate basic language abilities, the higher-order linguistic macrostructures (e.g., topic development) that may reflect top-down, concept-driven cognitive abilities remain underexplored. These macrostructural patterns are crucial for NCD detection, yet challenging to quantify due to their abstract and complex nature. To bridge this gap, we propose two novel macrostructural approaches: (1) a Dynamic Topic Model (DTM) to track topic evolution over time, and (2) a Text-Image Temporal Alignment Network (TITAN) to measure cross-modal consistency between narrative and visual stimuli. Experimental results show the effectiveness of the proposed approaches in NCD detection, with TITAN achieving superior performance across three corpora: ADReSS (F1 = 0.8889), ADReSSo (F1 = 0.8504), and CU-MARVEL-RABBIT (F1 = 0.7238). Feature contribution analysis reveals that macrostructural features (e.g., topic variability, topic change rate, and topic consistency) constitute the most significant contributors to the model's decision pathways, outperforming the investigated microstructural features. These findings underscore the value of macrostructural analysis for understanding linguistic-cognitive interactions associated with NCDs.","PeriodicalId":13038,"journal":{"name":"IEEE Journal of Selected Topics in Signal Processing","volume":"19 5","pages":"741-756"},"PeriodicalIF":13.7,"publicationDate":"2025-10-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145659183","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Robust MRI Reconstruction by Smoothed Unrolling (SMUG) 基于平滑展开（SMUG）的鲁棒MRI重建

IF 13.7 1区工程技术

IEEE Journal of Selected Topics in Signal Processing Pub Date : 2025-10-10 DOI: 10.1109/JSTSP.2025.3615540

Shijun Liang;Van Hoang Minh Nguyen;Jinghan Jia;Ismail R. Alkhouri;Sijia Liu;Saiprasad Ravishankar

{"title":"Robust MRI Reconstruction by Smoothed Unrolling (SMUG)","authors":"Shijun Liang;Van Hoang Minh Nguyen;Jinghan Jia;Ismail R. Alkhouri;Sijia Liu;Saiprasad Ravishankar","doi":"10.1109/JSTSP.2025.3615540","DOIUrl":"https://doi.org/10.1109/JSTSP.2025.3615540","url":null,"abstract":"As the popularity of deep learning (DL) in the field of magnetic resonance imaging (MRI) continues to rise, recent research has indicated that DL-based MRI reconstruction models might be excessively sensitive to minor input disturbances, including worst-case or random additive perturbations. This sensitivity often leads to unstable aliased images. This raises the question of how to devise DL techniques for MRI reconstruction that can be robust to these variations. To address this problem, we propose a novel image reconstruction framework, termed <sc>Smoothed Unrolling (<sc>SMUG), which advances a deep unrolling-based MRI reconstruction model using a randomized smoothing (RS)-based robust learning approach. RS, which improves the tolerance of a model against input noise, has been widely used in the design of adversarial defense approaches for image classification tasks. Yet, we find that the conventional design that applies RS to the entire DL-based MRI model is ineffective. In this paper, we show that <sc>SMUG and its variants address the above issue by customizing the RS process based on the unrolling architecture of DL-based MRI reconstruction models. We theoretically analyze the robustness of our method in the presence of perturbations. Compared to vanilla RS and other recent approaches, we show that <sc>SMUG improves the robustness of MRI reconstruction with respect to a diverse set of instability sources, including worst-case and random noise perturbations to input measurements, varying measurement sampling rates, and different numbers of unrolling steps.","PeriodicalId":13038,"journal":{"name":"IEEE Journal of Selected Topics in Signal Processing","volume":"19 7","pages":"1558-1573"},"PeriodicalIF":13.7,"publicationDate":"2025-10-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145859871","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

XPPG-PCA: Reference-Free Automatic Speech Severity Evaluation With Principal Components 基于主成分的无参考语音严重程度自动评估

IF 13.7 1区工程技术

IEEE Journal of Selected Topics in Signal Processing Pub Date : 2025-10-03 DOI: 10.1109/JSTSP.2025.3617859

Bence Mark Halpern;Thomas B. Tienkamp;Teja Rebernik;Rob J.J.H. van Son;Sebastiaan A.H.J. de Visscher;Max J.H. Witjes;Defne Abur;Tomoki Toda

{"title":"XPPG-PCA: Reference-Free Automatic Speech Severity Evaluation With Principal Components","authors":"Bence Mark Halpern;Thomas B. Tienkamp;Teja Rebernik;Rob J.J.H. van Son;Sebastiaan A.H.J. de Visscher;Max J.H. Witjes;Defne Abur;Tomoki Toda","doi":"10.1109/JSTSP.2025.3617859","DOIUrl":"https://doi.org/10.1109/JSTSP.2025.3617859","url":null,"abstract":"Reliably evaluating the severity of a speech pathology is crucial in healthcare. However, the current reliance on expert evaluations by speech-language pathologists presents several challenges: while their assessments are highly skilled, they are also subjective, time-consuming, and costly, which can limit the reproducibility of clinical studies and place a strain on healthcare resources. While automated methods exist, they have significant drawbacks. Reference-based approaches require transcriptions or healthy speech samples, restricting them to read speech and limiting their applicability. Existing reference-free methods are also flawed; supervised models often learn spurious shortcuts from data, while handcrafted features are often unreliable and restricted to specific speech tasks. This paper introduces XPPG-PCA (x-vector phonetic posteriorgram principal component analysis), a novel, unsupervised, reference-free method for speech severity evaluation. Using three Dutch oral cancer datasets, we demonstrate that XPPG-PCA performs comparably to, or exceeds established reference-based methods. Our experiments confirm its robustness against data shortcuts and noise, showing its potential for real-world clinical use. Taken together, our results show that XPPG-PCA provides a robust, generalizable solution for the objective assessment of speech pathology, with the potential to significantly improve the efficiency and reliability of clinical evaluations across a range of disorders. An open-source implementation is available.","PeriodicalId":13038,"journal":{"name":"IEEE Journal of Selected Topics in Signal Processing","volume":"19 5","pages":"783-795"},"PeriodicalIF":13.7,"publicationDate":"2025-10-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145659190","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Learning Against Distributional Uncertainty: On the Trade-Off Between Robustness and Specificity 针对分布不确定性的学习：关于稳健性与特异性之间的权衡

IF 8.5 1区工程技术

IEEE Journal of Selected Topics in Signal Processing Pub Date : 2025-10-01 Epub Date: 2025-03-05 DOI: 10.1109/JSTSP.2025.3567002

Shixiong Wang;Haowei Wang;Xinke Li;Jean Honorio

{"title":"Learning Against Distributional Uncertainty: On the Trade-Off Between Robustness and Specificity","authors":"Shixiong Wang;Haowei Wang;Xinke Li;Jean Honorio","doi":"10.1109/JSTSP.2025.3567002","DOIUrl":"https://doi.org/10.1109/JSTSP.2025.3567002","url":null,"abstract":"Trustworthy machine learning aims at combating distributional uncertainties in training data distributions compared to population distributions. Typical treatment frameworks include the Bayesian approach, (min-max) distributionally robust optimization (DRO), and regularization. However, three issues have to be raised: 1) the prior distribution in the Bayesian method and the regularizer in the regularization method are difficult to specify; 2) the DRO method tends to be overly conservative; 3) all the three methods are biased estimators of the true optimal cost. This paper studies a new framework that unifies the three approaches and addresses the three challenges above. The asymptotic properties (e.g., consistencies and asymptotic normalities), non-asymptotic properties (e.g., generalization bounds and unbiasedness), and solution methods of the proposed model are studied. The new model reveals the trade-off between the robustness to the unseen data and the specificity to the training data. Experiments on various real-world tasks validate the superiority of the proposed learning framework.","PeriodicalId":13038,"journal":{"name":"IEEE Journal of Selected Topics in Signal Processing","volume":"19 7","pages":"1420-1435"},"PeriodicalIF":8.5,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148511996","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

PLEX: Perturbation-Free Local Explanations for LLM-Based Text Classification 基于llm的文本分类的无扰动局部解释

IF 8.5 1区工程技术

IEEE Journal of Selected Topics in Signal Processing Pub Date : 2025-10-01 Epub Date: 2025-11-19 DOI: 10.1109/JSTSP.2025.3633593

Yogachandran Rahulamathavan;Misbah Farooq;Varuna De Silva

{"title":"PLEX: Perturbation-Free Local Explanations for LLM-Based Text Classification","authors":"Yogachandran Rahulamathavan;Misbah Farooq;Varuna De Silva","doi":"10.1109/JSTSP.2025.3633593","DOIUrl":"https://doi.org/10.1109/JSTSP.2025.3633593","url":null,"abstract":"Large Language Models (LLMs) excel in text classification, but their complexity hinders interpretability, making it difficult to understand the reasoning behind their predictions. Explainable AI (XAI) methods like LIME and SHAP offer local explanations by identifying influential words, but they rely on computationally expensive perturbations. These methods typically generate thousands of perturbed sentences and perform inferences on each, incurring a substantial computational burden, especially with LLMs. To address this, we propose <underline>Perturbation-free <underline>Local <underline>Explanation (PLEX), a novel method that leverages the contextual embeddings extracted from the LLM and a “Siamese network” style neural network trained to align with feature importance scores. This one-off training eliminates the need for subsequent perturbations, enabling efficient explanations for any new sentence. We demonstrate PLEX's effectiveness on four different classification tasks (sentiment, fake news, fake COVID-19 news and depression), showing more than 92% agreement with LIME and SHAP. Our evaluation using a “stress test” reveals that PLEX accurately identifies influential words, leading to a similar decline in classification accuracy as observed with LIME and SHAP when these words are removed. Notably, in some cases, PLEX demonstrates superior performance in capturing the impact of key features. PLEX dramatically accelerates explanation, reducing time and computational overhead by two and four orders of magnitude, respectively. This work offers a promising solution for explainable LLM-based text classification.","PeriodicalId":13038,"journal":{"name":"IEEE Journal of Selected Topics in Signal Processing","volume":"19 7","pages":"1266-1278"},"PeriodicalIF":8.5,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148513124","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

A Review of Faithfulness Metrics for Hallucination Assessment in Large Language Models 大型语言模型幻觉评估的忠实度指标综述

IF 8.5 1区工程技术

IEEE Journal of Selected Topics in Signal Processing Pub Date : 2025-10-01 Epub Date: 2025-06-12 DOI: 10.1109/JSTSP.2025.3579203

Ben Malin;Tatiana Kalganova;Nikolaos Boulgouris

引用次数: 0

Co-Learning-Aided Multi-Modal-Deep-Learning Framework of Passive DOA Estimators for a Heterogeneous Hybrid Massive MIMO Receiver 异构混合海量MIMO接收机被动DOA估计的协同学习辅助多模深度学习框架

IF 8.5 1区工程技术

IEEE Journal of Selected Topics in Signal Processing Pub Date : 2025-10-01 Epub Date: 2025-10-27 DOI: 10.1109/JSTSP.2025.3626260

Jiatong Bai;Feng Shu;Fuhui Zhou;Qinghe Zheng;Bo Xu;Baihua Shi;Yiwen Chen;Weibin Zhang;Xianpeng Wang

{"title":"Co-Learning-Aided Multi-Modal-Deep-Learning Framework of Passive DOA Estimators for a Heterogeneous Hybrid Massive MIMO Receiver","authors":"Jiatong Bai;Feng Shu;Fuhui Zhou;Qinghe Zheng;Bo Xu;Baihua Shi;Yiwen Chen;Weibin Zhang;Xianpeng Wang","doi":"10.1109/JSTSP.2025.3626260","DOIUrl":"https://doi.org/10.1109/JSTSP.2025.3626260","url":null,"abstract":"Due to its excellent performance in rate and resolution, fully-digital (FD) massive multiple-input multiple-output (MIMO) antenna arrays has been widely applied in data transmission and direction of arrival (DOA) measurements, etc. But it confronts with two main challenges: high computational complexity and circuit cost. The two problems may be addressed well by hybrid analog-digital (HAD) structure. But there exists the problem of phase ambiguity for HAD, which leads to its low-efficiency or high-latency. Does exist there such a MIMO structure of owning low-cost, low-complexity and high time efficiency at the same time. To satisfy the three properties, a novel heterogeneous hybrid MIMO receiver structure of integrating FD and heterogeneous HAD (<inline-formula><tex-math>$rm {H}^{2}$</tex-math></inline-formula>AD-FD) is proposed and corresponding multi-modal (MD)-learning framework is developed. The framework includes three major stages: 1) generate the candidate sets via root multiple signal classification (Root-MUSIC) or deep learning (DL); 2) infer the class of true solutions from candidate sets using machine learning (ML) methods; 3) fuse the two-part true solutions to achieve a better DOA estimation. The above process form two methods named MD-Root-MUSIC and MDDL. To improve DOA estimation accuracy and reduce the clustering complexity, a co-learning-aided MD framework is proposed to form two enhanced methods named CoMDDL and CoMD-RootMUSIC. Moreover, the Cramer-Rao lower bound (CRLB) for the proposed <inline-formula><tex-math>$rm {H}^{2}$</tex-math></inline-formula>AD-FD structure is also derived. Experimental results demonstrate that our proposed four methods could approach the CRLB for signal-to-noise ratio (SNR) > 0 dB and the proposed CoMDDL and MDDL perform better than CoMD-RootMUSIC and MD-RootMUSIC, particularly in the extremely low SNR region.","PeriodicalId":13038,"journal":{"name":"IEEE Journal of Selected Topics in Signal Processing","volume":"19 7","pages":"1448-1460"},"PeriodicalIF":8.5,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148511939","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Split Fine-Tuning for Large Language Models in Wireless Networks 无线网络中大型语言模型的分割微调

IF 8.5 1区工程技术

IEEE Journal of Selected Topics in Signal Processing Pub Date : 2025-10-01 Epub Date: 2025-06-19 DOI: 10.1109/JSTSP.2025.3581484

Songge Zhang;Guoliang Cheng;Wen Wu;Xinyu Huang;Lingyang Song;Xuemin Shen

{"title":"Split Fine-Tuning for Large Language Models in Wireless Networks","authors":"Songge Zhang;Guoliang Cheng;Wen Wu;Xinyu Huang;Lingyang Song;Xuemin Shen","doi":"10.1109/JSTSP.2025.3581484","DOIUrl":"https://doi.org/10.1109/JSTSP.2025.3581484","url":null,"abstract":"Fine-tuning is the process of adapting the pre-trained large language models (LLMs) for downstream tasks. Due to substantial model parameters, fine-tuning LLM on mobile devices demands considerable memory resources, and suffers from high communication overhead and fine-tuning delay. In this paper, we propose an efficient LLM fine-tuning scheme in wireless networks, named <underline>Split <underline>Fine-<underline>Tuning (SFT), which can accommodate LLM fine-tuning on mobile devices. Specifically, an LLM is split into a server-side part on the edge server and a device-side part on the mobile device to satisfy the device-side memory constraint. All devices share a server-side model and perform parallel fine-tuning to reduce fine-tuning delay. In addition, to reduce communication overhead incurred by data exchange between devices and the edge server, we propose an activation data compression scheme by jointly leveraging sparsification, stochastic quantization, and lossless encoding methods. Furthermore, we formulate a fine-tuning delay minimization problem under model accuracy and device-side memory constraints, taking device heterogeneity and channel dynamics into account. To solve the problem, the nonlinear mixed-integer problem is decoupled into two subproblems in different timescales. A two-timescale resource management algorithm is proposed to jointly optimize the compression rate and transformer block allocation in the large timescale using the augmented Lagrangian method, and determine spectrum resource allocation in the small timescale via sequential quadratic programming. Extensive simulation results demonstrate that the proposed scheme can reduce the fine-tuning delay by up to 66.4% and communication overhead by 93.6% compared to state-of-the-art benchmarks, while satisfying device-side memory and model accuracy constraints.","PeriodicalId":13038,"journal":{"name":"IEEE Journal of Selected Topics in Signal Processing","volume":"19 7","pages":"1376-1391"},"PeriodicalIF":8.5,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148511902","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Measuring Training Variability From Stochastic Optimization Using Robust Nonparametric Testing 用鲁棒非参数检验测量随机优化训练的可变性

IF 8.5 1区工程技术

IEEE Journal of Selected Topics in Signal Processing Pub Date : 2025-10-01 Epub Date: 2025-06-27 DOI: 10.1109/JSTSP.2025.3583140

Sinjini Banerjee;Tim Marrinan;Reilly Cannon;Tony Chiang;Anand D. Sarwate

{"title":"Measuring Training Variability From Stochastic Optimization Using Robust Nonparametric Testing","authors":"Sinjini Banerjee;Tim Marrinan;Reilly Cannon;Tony Chiang;Anand D. Sarwate","doi":"10.1109/JSTSP.2025.3583140","DOIUrl":"https://doi.org/10.1109/JSTSP.2025.3583140","url":null,"abstract":"Deep neural network training often involves stochastic optimization, meaning each run will produce a different model. This implies that hyperparameters of the training process, such as the random seed itself, can potentially have a significant influence on the variability in the trained models. Measuring model quality by summary statistics, such as test accuracy, can obscure this dependence. We propose a robust hypothesis testing framework and a novel summary statistic, the <inline-formula><tex-math>$alpha$</tex-math></inline-formula>-trimming level, to measure model similarity. Applying hypothesis testing directly with the <inline-formula><tex-math>$alpha$</tex-math></inline-formula>-trimming level is challenging because we cannot accurately describe the distribution under the null hypothesis. Our framework addresses this issue by determining how closely an approximate distribution resembles the expected distribution of a group of individually trained models and using this approximation as our reference. We then use the <inline-formula><tex-math>$alpha$</tex-math></inline-formula>-trimming level to suggest how many training runs should be sampled to ensure that an ensemble is a reliable representative of the true model performance. We also show how to use the <inline-formula><tex-math>$alpha$</tex-math></inline-formula>-trimming level to measure model variability and demonstrate experimentally that it is more expressive than performance metrics like validation accuracy, churn, or expected calibration error when taken alone. An application of fine-tuning over random seed in transfer learning illustrates the advantage of our new metric.","PeriodicalId":13038,"journal":{"name":"IEEE Journal of Selected Topics in Signal Processing","volume":"19 7","pages":"1477-1492"},"PeriodicalIF":8.5,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148513123","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0