IEEE journal on selected areas in information theory最新文献_第3页

Learning Algorithm Generalization Error Bounds via Auxiliary Distributions 通过辅助分布实现学习算法泛化误差边界

IEEE journal on selected areas in information theory Pub Date : 2024-04-25 DOI: 10.1109/JSAIT.2024.3391900

Gholamali Aminian;Saeed Masiha;Laura Toni;Miguel R. D. Rodrigues

{"title":"Learning Algorithm Generalization Error Bounds via Auxiliary Distributions","authors":"Gholamali Aminian;Saeed Masiha;Laura Toni;Miguel R. D. Rodrigues","doi":"10.1109/JSAIT.2024.3391900","DOIUrl":"https://doi.org/10.1109/JSAIT.2024.3391900","url":null,"abstract":"Generalization error bounds are essential for comprehending how well machine learning models work. In this work, we suggest a novel method, i.e., the Auxiliary Distribution Method, that leads to new upper bounds on expected generalization errors that are appropriate for supervised learning scenarios. We show that our general upper bounds can be specialized under some conditions to new bounds involving the \u0000<inline-formula> <tex-math>$alpha $ </tex-math></inline-formula>\u0000-Jensen-Shannon, \u0000<inline-formula> <tex-math>$alpha $ </tex-math></inline-formula>\u0000-Rényi \u0000<inline-formula> <tex-math>$(0lt alpha lt 1)$ </tex-math></inline-formula>\u0000 information between a random variable modeling the set of training samples and another random variable modeling the set of hypotheses. Our upper bounds based on \u0000<inline-formula> <tex-math>$alpha $ </tex-math></inline-formula>\u0000-Jensen-Shannon information are also finite. Additionally, we demonstrate how our auxiliary distribution method can be used to derive the upper bounds on excess risk of some learning algorithms in the supervised learning context and the generalization error under the distribution mismatch scenario in supervised learning algorithms, where the distribution mismatch is modeled as \u0000<inline-formula> <tex-math>$alpha $ </tex-math></inline-formula>\u0000-Jensen-Shannon or \u0000<inline-formula> <tex-math>$alpha $ </tex-math></inline-formula>\u0000-Rényi divergence between the distribution of test and training data samples distributions. We also outline the conditions for which our proposed upper bounds might be tighter than other earlier upper bounds.","PeriodicalId":73295,"journal":{"name":"IEEE journal on selected areas in information theory","volume":"5 ","pages":"273-284"},"PeriodicalIF":0.0,"publicationDate":"2024-04-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141096301","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Neural Distributed Compressor Discovers Binning 神经分布式压缩器发现分选功能

IEEE journal on selected areas in information theory Pub Date : 2024-04-24 DOI: 10.1109/JSAIT.2024.3393429

Ezgi Ozyilkan;Johannes Ballé;Elza Erkip

引用次数: 0

Training Generative Models From Privatized Data via Entropic Optimal Transport 通过熵优化传输从私有化数据中训练生成模型

IEEE journal on selected areas in information theory Pub Date : 2024-04-16 DOI: 10.1109/JSAIT.2024.3387463

Daria Reshetova;Wei-Ning Chen;Ayfer Özgür

引用次数: 0

Differentially Private Stochastic Linear Bandits: (Almost) for Free 差分私有随机线性强盗：（几乎）免费

IEEE journal on selected areas in information theory Pub Date : 2024-04-16 DOI: 10.1109/JSAIT.2024.3389954

Osama Hanna;Antonious M. Girgis;Christina Fragouli;Suhas Diggavi

{"title":"Differentially Private Stochastic Linear Bandits: (Almost) for Free","authors":"Osama Hanna;Antonious M. Girgis;Christina Fragouli;Suhas Diggavi","doi":"10.1109/JSAIT.2024.3389954","DOIUrl":"https://doi.org/10.1109/JSAIT.2024.3389954","url":null,"abstract":"In this paper, we propose differentially private algorithms for the problem of stochastic linear bandits in the central, local and shuffled models. In the central model, we achieve almost the same regret as the optimal non-private algorithms, which means we get privacy for free. In particular, we achieve a regret of \u0000<inline-formula> <tex-math>$tilde {O}left({sqrt {T}+{}frac {1}{varepsilon }}right)$ </tex-math></inline-formula>\u0000 matching the known lower bound for private linear bandits, while the best previously known algorithm achieves \u0000<inline-formula> <tex-math>$tilde {O}left({{}frac {1}{varepsilon }sqrt {T}}right)$ </tex-math></inline-formula>\u0000. In the local case, we achieve a regret of \u0000<inline-formula> <tex-math>$tilde {O}left({{}frac {1}{varepsilon }{sqrt {T}}}right)$ </tex-math></inline-formula>\u0000 which matches the non-private regret for constant \u0000<inline-formula> <tex-math>$varepsilon $ </tex-math></inline-formula>\u0000, but suffers a regret penalty when \u0000<inline-formula> <tex-math>$varepsilon $ </tex-math></inline-formula>\u0000 is small. In the shuffled model, we also achieve regret of \u0000<inline-formula> <tex-math>$tilde {O}left({sqrt {T}+{}frac {1}{varepsilon }}right)$ </tex-math></inline-formula>\u0000 while the best previously known algorithm suffers a regret of \u0000<inline-formula> <tex-math>$tilde {O}left({{}frac {1}{varepsilon }{T^{3/5}}}right)$ </tex-math></inline-formula>\u0000. Our numerical evaluation validates our theoretical results. Our results generalize for contextual linear bandits with known context distributions.","PeriodicalId":73295,"journal":{"name":"IEEE journal on selected areas in information theory","volume":"5 ","pages":"135-147"},"PeriodicalIF":0.0,"publicationDate":"2024-04-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140818731","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Improving Group Testing via Gradient Descent 通过梯度下降改进分组测试

IEEE journal on selected areas in information theory Pub Date : 2024-04-15 DOI: 10.1109/JSAIT.2024.3386182

Sundara Rajan Srinivasavaradhan;Pavlos Nikolopoulos;Christina Fragouli;Suhas Diggavi

引用次数: 0

Optimal Binary Differential Privacy via Graphs 通过图形实现最佳二进制差分隐私保护

IEEE journal on selected areas in information theory Pub Date : 2024-04-11 DOI: 10.1109/JSAIT.2024.3384183

Sahel Torkamani;Javad B. Ebrahimi;Parastoo Sadeghi;Rafael G. L. D’Oliveira;Muriel Médard

{"title":"Optimal Binary Differential Privacy via Graphs","authors":"Sahel Torkamani;Javad B. Ebrahimi;Parastoo Sadeghi;Rafael G. L. D’Oliveira;Muriel Médard","doi":"10.1109/JSAIT.2024.3384183","DOIUrl":"https://doi.org/10.1109/JSAIT.2024.3384183","url":null,"abstract":"We present the notion of \u0000<italic>reasonable utility</i>\u0000 for binary mechanisms, which applies to all utility functions in the literature. This notion induces a partial ordering on the performance of all binary differentially private (DP) mechanisms. DP mechanisms that are maximal elements of this ordering are optimal DP mechanisms for every reasonable utility. By looking at differential privacy as a randomized graph coloring, we characterize these optimal DP in terms of their behavior on a certain subset of the boundary datasets we call a boundary hitting set. In the process of establishing our results, we also introduce a useful notion that generalizes DP conditions for binary-valued queries, which we coin as suitable pairs. Suitable pairs abstract away the algebraic roles of \u0000<inline-formula> <tex-math>$varepsilon ,delta $ </tex-math></inline-formula>\u0000 in the DP framework, making the derivations and understanding of our proofs simpler. Additionally, the notion of a suitable pair can potentially capture privacy conditions in frameworks other than DP and may be of independent interest.","PeriodicalId":73295,"journal":{"name":"IEEE journal on selected areas in information theory","volume":"5 ","pages":"162-174"},"PeriodicalIF":0.0,"publicationDate":"2024-04-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140818803","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Iterative Sketching for Secure Coded Regression 安全编码回归的迭代草图绘制

IEEE journal on selected areas in information theory Pub Date : 2024-04-04 DOI: 10.1109/JSAIT.2024.3384395

Neophytos Charalambides;Hessam Mahdavifar;Mert Pilanci;Alfred O. Hero

引用次数: 0

Total Variation Meets Differential Privacy 总变化符合差异隐私

IEEE journal on selected areas in information theory Pub Date : 2024-04-04 DOI: 10.1109/JSAIT.2024.3384083

Elena Ghazi;Ibrahim Issa

引用次数: 0

The Worst-Case Data-Generating Probability Measure in Statistical Learning 统计学习中的最坏情况数据生成概率度量

IEEE journal on selected areas in information theory Pub Date : 2024-04-02 DOI: 10.1109/JSAIT.2024.3383281

Xinying Zou;Samir M. Perlaza;Iñaki Esnaola;Eitan Altman;H. Vincent Poor

{"title":"The Worst-Case Data-Generating Probability Measure in Statistical Learning","authors":"Xinying Zou;Samir M. Perlaza;Iñaki Esnaola;Eitan Altman;H. Vincent Poor","doi":"10.1109/JSAIT.2024.3383281","DOIUrl":"https://doi.org/10.1109/JSAIT.2024.3383281","url":null,"abstract":"The worst-case data-generating (WCDG) probability measure is introduced as a tool for characterizing the generalization capabilities of machine learning algorithms. Such a WCDG probability measure is shown to be the unique solution to two different optimization problems: \u0000<inline-formula> <tex-math>$(a)$ </tex-math></inline-formula>\u0000 The maximization of the expected loss over the set of probability measures on the datasets whose relative entropy with respect to a \u0000<italic>reference measure</i>\u0000 is not larger than a given threshold; and \u0000<inline-formula> <tex-math>$(b)$ </tex-math></inline-formula>\u0000 The maximization of the expected loss with regularization by relative entropy with respect to the reference measure. Such a reference measure can be interpreted as a prior on the datasets. The WCDG cumulants are finite and bounded in terms of the cumulants of the reference measure. To analyze the concentration of the expected empirical risk induced by the WCDG probability measure, the notion of \u0000<inline-formula> <tex-math>$(epsilon, delta )$ </tex-math></inline-formula>\u0000-robustness of models is introduced. Closed-form expressions are presented for the sensitivity of the expected loss for a fixed model. These results lead to a novel expression for the generalization error of arbitrary machine learning algorithms. This exact expression is provided in terms of the WCDG probability measure and leads to an upper bound that is equal to the sum of the mutual information and the lautum information between the models and the datasets, up to a constant factor. This upper bound is achieved by a Gibbs algorithm. This finding reveals that an exploration into the generalization error of the Gibbs algorithm facilitates the derivation of overarching insights applicable to any machine learning algorithm.","PeriodicalId":73295,"journal":{"name":"IEEE journal on selected areas in information theory","volume":"5 ","pages":"175-189"},"PeriodicalIF":0.0,"publicationDate":"2024-04-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140844521","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

On the Computation of the Gaussian Rate–Distortion–Perception Function 关于高斯速率-失真-感知函数的计算

IEEE journal on selected areas in information theory Pub Date : 2024-03-29 DOI: 10.1109/JSAIT.2024.3381230

Giuseppe Serra;Photios A. Stavrou;Marios Kountouris

{"title":"On the Computation of the Gaussian Rate–Distortion–Perception Function","authors":"Giuseppe Serra;Photios A. Stavrou;Marios Kountouris","doi":"10.1109/JSAIT.2024.3381230","DOIUrl":"https://doi.org/10.1109/JSAIT.2024.3381230","url":null,"abstract":"In this paper, we study the computation of the rate-distortion-perception function (RDPF) for a multivariate Gaussian source assuming jointly Gaussian reconstruction under mean squared error (MSE) distortion and, respectively, Kullback–Leibler divergence, geometric Jensen-Shannon divergence, squared Hellinger distance, and squared Wasserstein-2 distance perception metrics. To this end, we first characterize the analytical bounds of the scalar Gaussian RDPF for the aforementioned divergence functions, also providing the RDPF-achieving forward “test-channel” realization. Focusing on the multivariate case, assuming jointly Gaussian reconstruction and tensorizable distortion and perception metrics, we establish that the optimal solution resides on the vector space spanned by the eigenvector of the source covariance matrix. Consequently, the multivariate optimization problem can be expressed as a function of the scalar Gaussian RDPFs of the source marginals, constrained by global distortion and perception levels. Leveraging this characterization, we design an alternating minimization scheme based on the block nonlinear Gauss–Seidel method, which optimally solves the problem while identifying the Gaussian RDPF-achieving realization. Furthermore, the associated algorithmic embodiment is provided, as well as the convergence and the rate of convergence characterization. Lastly, for the “perfect realism” regime, the analytical solution for the multivariate Gaussian RDPF is obtained. We corroborate our results with numerical simulations and draw connections to existing results.","PeriodicalId":73295,"journal":{"name":"IEEE journal on selected areas in information theory","volume":"5 ","pages":"314-330"},"PeriodicalIF":0.0,"publicationDate":"2024-03-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141084836","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0