Doklady Mathematics最新文献_第7页

Activations and Gradients Compression for Model-Parallel Training 用于模型并行训练的激活和梯度压缩

IF 0.5 4区数学

Doklady Mathematics Pub Date : 2024-03-25 DOI: 10.1134/S1064562423701314

M. I. Rudakov, A. N. Beznosikov, Ya. A. Kholodov, A. V. Gasnikov

{"title":"Activations and Gradients Compression for Model-Parallel Training","authors":"M. I. Rudakov, A. N. Beznosikov, Ya. A. Kholodov, A. V. Gasnikov","doi":"10.1134/S1064562423701314","DOIUrl":"10.1134/S1064562423701314","url":null,"abstract":"Large neural networks require enormous computational clusters of machines. Model-parallel training, when the model architecture is partitioned sequentially between workers, is a popular approach for training modern models. Information compression can be applied to decrease workers’ communication time, as it is often a bottleneck in such systems. This work explores how simultaneous compression of activations and gradients in model-parallel distributed training setup affects convergence. We analyze compression methods such as quantization and TopK compression, and also experiment with error compensation techniques. Moreover, we employ TopK with AQ-SGD per-batch error feedback approach. We conduct experiments on image classification and language model fine-tuning tasks. Our findings demonstrate that gradients require milder compression rates than activations. We observe that (K = 10% ) is the lowest TopK compression level, which does not harm model convergence severely. Experiments also show that models trained with TopK perform well only when compression is also applied during inference. We find that error feedback techniques do not improve model-parallel training compared to plain compression, but allow model inference without compression with almost no quality drop. Finally, when applied with the AQ-SGD approach, TopK stronger than with (K = 30% ) worsens model performance significantly.","PeriodicalId":531,"journal":{"name":"Doklady Mathematics","volume":"108 2 supplement","pages":"S272 - S281"},"PeriodicalIF":0.5,"publicationDate":"2024-03-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142413765","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Neural Network Approach to the Problem of Predicting Interest Rate Anomalies under the Influence of Correlated Noise 预测相关噪声影响下利率异常问题的神经网络方法

IF 0.5 4区数学

Doklady Mathematics Pub Date : 2024-03-25 DOI: 10.1134/S1064562423701521

G. A. Zotov, P. P. Lukianchenko

引用次数: 0

Do we Benefit from the Categorization of the News Flow in the Stock Price Prediction Problem? 在股价预测问题中，我们是否能从新闻流的分类中获益？

IF 0.5 4区数学

Doklady Mathematics Pub Date : 2024-03-25 DOI: 10.1134/S1064562423701648

T. D. Kulikova, E. Yu. Kovtun, S. A. Budennyy

{"title":"Do we Benefit from the Categorization of the News Flow in the Stock Price Prediction Problem?","authors":"T. D. Kulikova, E. Yu. Kovtun, S. A. Budennyy","doi":"10.1134/S1064562423701648","DOIUrl":"10.1134/S1064562423701648","url":null,"abstract":"The power of machine learning is widely leveraged in the task of company stock price prediction. It is essential to incorporate historical stock prices and relevant external world information for constructing a more accurate predictive model. The sentiments of the financial news connected with the company can become such valuable knowledge. However, financial news has different topics, such as Macro, Markets, or Product news. The adoption of such categorization is usually out of scope in a market research. In this work, we aim to close this gap and explore the effect of capturing the news topic differentiation in the stock price prediction problem. Initially, we classify the financial news stream into 20 pre-defined topics with the pre-trained model. Then, we get sentiments and explore the topic of news group sentiment labeling. Moreover, we conduct the experiments with the several well-proved models for time series forecasting, including the Temporal Convolutional Network (TCN), the D-Linear, the Transformer, and the Temporal Fusion Transformer (TFT). In the results of our research, utilizing the information from separate topic groups contributes to a better performance of deep learning models compared to the approach when we consider all news sentiments without any division.","PeriodicalId":531,"journal":{"name":"Doklady Mathematics","volume":"108 2 supplement","pages":"S503 - S510"},"PeriodicalIF":0.5,"publicationDate":"2024-03-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140884599","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Machine Learning As a Tool to Accelerate the Search for New Materials for Metal-Ion Batteries 将机器学习作为加速寻找金属离子电池新材料的工具

IF 0.5 4区数学

Doklady Mathematics Pub Date : 2024-03-25 DOI: 10.1134/S1064562423701612

V. T. Osipov, M. I. Gongola, Ye. A. Morkhova, A. P. Nemudryi, A. A. Kabanov

引用次数: 0

Statistical Online Learning in Recurrent and Feedforward Quantum Neural Networks 递归和前馈量子神经网络中的统计在线学习

IF 0.5 4区数学

Doklady Mathematics Pub Date : 2024-03-25 DOI: 10.1134/S1064562423701557

S. V. Zuev

{"title":"Statistical Online Learning in Recurrent and Feedforward Quantum Neural Networks","authors":"S. V. Zuev","doi":"10.1134/S1064562423701557","DOIUrl":"10.1134/S1064562423701557","url":null,"abstract":"For adaptive artificial intelligence systems, the question of the possibility of online learning is especially important, since such training provides adaptation. The purpose of the work is to consider methods of quantum machine online learning for the two most common architectures of quantum neural networks: feedforward and recurrent. The work uses the quantumz module available on PyPI to emulate quantum computing and create artificial quantum neural networks. In addition, the genser module is used to transform data dimensions, which provides reversible transformation of dimensions without loss of information. The data for the experiments are taken from open sources. The paper implements the machine learning method without optimization, proposed by the author earlier. Online learning algorithms for recurrent and feedforward quantum neural network are presented and experimentally confirmed. The proposed learning algorithms can be used as data science tools, as well as a part of adaptive intelligent control systems. The developed software can fully unleash its potential only on quantum computers, but, in the case of a small number of quantum registers, it can also be used in systems that emulate quantum computing, or in photonic computers.","PeriodicalId":531,"journal":{"name":"Doklady Mathematics","volume":"108 2 supplement","pages":"S317 - S324"},"PeriodicalIF":0.5,"publicationDate":"2024-03-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142413768","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

MTS Kion Implicit Contextualised Sequential Dataset for Movie Recommendation 用于电影推荐的 MTS Kion 隐含语境化序列数据集

IF 0.5 4区数学

Doklady Mathematics Pub Date : 2024-03-25 DOI: 10.1134/S1064562423701594

I. Safilo, D. Tikhonovich, A. V. Petrov, D. I. Ignatov

引用次数: 0

Optimal Data Splitting in Distributed Optimization for Machine Learning 机器学习分布式优化中的最佳数据分割

IF 0.5 4区数学

Doklady Mathematics Pub Date : 2024-03-25 DOI: 10.1134/S1064562423701600

D. Medyakov, G. Molodtsov, A. Beznosikov, A. Gasnikov

{"title":"Optimal Data Splitting in Distributed Optimization for Machine Learning","authors":"D. Medyakov, G. Molodtsov, A. Beznosikov, A. Gasnikov","doi":"10.1134/S1064562423701600","DOIUrl":"10.1134/S1064562423701600","url":null,"abstract":"The distributed optimization problem has become increasingly relevant recently. It has a lot of advantages such as processing a large amount of data in less time compared to non-distributed methods. However, most distributed approaches suffer from a significant bottleneck—the cost of communications. Therefore, a large amount of research has recently been directed at solving this problem. One such approach uses local data similarity. In particular, there exists an algorithm provably optimally exploiting the similarity property. But this result, as well as results from other works solve the communication bottleneck by focusing only on the fact that communication is significantly more expensive than local computing and does not take into account the various capacities of network devices and the different relationship between communication time and local computing expenses. We consider this setup and the objective of this study is to achieve an optimal ratio of distributed data between the server and local machines for any costs of communications and local computations. The running times of the network are compared between uniform and optimal distributions. The superior theoretical performance of our solutions is experimentally validated.","PeriodicalId":531,"journal":{"name":"Doklady Mathematics","volume":"108 2 supplement","pages":"S465 - S475"},"PeriodicalIF":0.5,"publicationDate":"2024-03-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140299620","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

1-Dimensional Topological Invariants to Estimate Loss Surface Non-Convexity 估算损失面非凸性的一维拓扑不变式

IF 0.5 4区数学

Doklady Mathematics Pub Date : 2024-03-25 DOI: 10.1134/S1064562423701569

D. S. Voronkova, S. A. Barannikov, E. V. Burnaev

引用次数: 0

Safe Pretraining of Deep Language Models in a Synthetic Pseudo-Language 在合成伪语言中安全预训练深度语言模型

IF 0.5 4区数学

Doklady Mathematics Pub Date : 2024-03-25 DOI: 10.1134/S1064562423701636

T. E. Gorbacheva, I. Y. Bondarenko

引用次数: 0

Optimal Analysis of Method with Batching for Monotone Stochastic Finite-Sum Variational Inequalities 单调随机有限和变分不等式的批处理方法优化分析