Dynamic & norm-based weights to normalize imbalance in back-propagated gradients of physics-informed neural networks

IF 1.1 Q3 PHYSICS, MULTIDISCIPLINARY
S. Deguchi, M. Asai
{"title":"Dynamic & norm-based weights to normalize imbalance in back-propagated gradients of physics-informed neural networks","authors":"S. Deguchi, M. Asai","doi":"10.1088/2399-6528/ace416","DOIUrl":null,"url":null,"abstract":"Physics-Informed Neural Networks (PINNs) have been a promising machine learning model for evaluating various physical problems. Despite their success in solving many types of partial differential equations (PDEs), some problems have been found to be difficult to learn, implying that the baseline PINNs is biased towards learning the governing PDEs while relatively neglecting given initial or boundary conditions. In this work, we propose Dynamically Normalized Physics-Informed Neural Networks (DN-PINNs), a method to train PINNs while evenly distributing multiple back-propagated gradient components. DN-PINNs determine the relative weights assigned to initial or boundary condition losses based on gradient norms, and the weights are updated dynamically during training. Through several numerical experiments, we demonstrate that DN-PINNs effectively avoids the imbalance in multiple gradients and improves the inference accuracy while keeping the additional computational cost within a reasonable range. Furthermore, we compare DN-PINNs with other PINNs variants and empirically show that DN-PINNs is competitive with or outperforms them. In addition, since DN-PINN uses exponential decay to update the relative weight, the weights obtained are biased toward the initial values. We study this initialization bias and show that a simple bias correction technique can alleviate this problem.","PeriodicalId":47089,"journal":{"name":"Journal of Physics Communications","volume":null,"pages":null},"PeriodicalIF":1.1000,"publicationDate":"2023-07-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Physics Communications","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1088/2399-6528/ace416","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"PHYSICS, MULTIDISCIPLINARY","Score":null,"Total":0}
引用次数: 0

Abstract

Physics-Informed Neural Networks (PINNs) have been a promising machine learning model for evaluating various physical problems. Despite their success in solving many types of partial differential equations (PDEs), some problems have been found to be difficult to learn, implying that the baseline PINNs is biased towards learning the governing PDEs while relatively neglecting given initial or boundary conditions. In this work, we propose Dynamically Normalized Physics-Informed Neural Networks (DN-PINNs), a method to train PINNs while evenly distributing multiple back-propagated gradient components. DN-PINNs determine the relative weights assigned to initial or boundary condition losses based on gradient norms, and the weights are updated dynamically during training. Through several numerical experiments, we demonstrate that DN-PINNs effectively avoids the imbalance in multiple gradients and improves the inference accuracy while keeping the additional computational cost within a reasonable range. Furthermore, we compare DN-PINNs with other PINNs variants and empirically show that DN-PINNs is competitive with or outperforms them. In addition, since DN-PINN uses exponential decay to update the relative weight, the weights obtained are biased toward the initial values. We study this initialization bias and show that a simple bias correction technique can alleviate this problem.
基于动态和范数的权重来归一化物理信息神经网络反向传播梯度中的不平衡
物理知情神经网络(PINN)是一种很有前途的机器学习模型,用于评估各种物理问题。尽管它们在求解许多类型的偏微分方程(PDE)方面取得了成功,但已经发现一些问题很难学习,这意味着基线PINN偏向于学习控制偏微分方程,而相对忽略给定的初始或边界条件。在这项工作中,我们提出了动态归一化物理知情神经网络(DN-PINN),这是一种在均匀分布多个反向传播梯度分量的情况下训练PINN的方法。DN PINN基于梯度范数确定分配给初始或边界条件损失的相对权重,并且在训练期间动态更新权重。通过几个数值实验,我们证明了DN-PINN有效地避免了多个梯度中的不平衡,提高了推理精度,同时将额外的计算成本保持在合理的范围内。此外,我们将DN PINN与其他PINN变体进行了比较,并从经验上表明DN PINN具有竞争力或优于它们。此外,由于DN-PINN使用指数衰减来更新相对权重,因此获得的权重偏向初始值。我们研究了这种初始化偏差,并表明一种简单的偏差校正技术可以缓解这个问题。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
Journal of Physics Communications
Journal of Physics Communications PHYSICS, MULTIDISCIPLINARY-
CiteScore
2.60
自引率
0.00%
发文量
114
审稿时长
10 weeks
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信